{"id":378,"date":"2019-12-20T15:04:17","date_gmt":"2019-12-20T23:04:17","guid":{"rendered":"http:\/\/blog.mozilla.org\/sfink\/?p=378"},"modified":"2020-01-06T13:51:17","modified_gmt":"2020-01-06T21:51:17","slug":"running-taskcluster-tasks-locally","status":"publish","type":"post","link":"https:\/\/blog.mozilla.org\/sfink\/2019\/12\/20\/running-taskcluster-tasks-locally\/","title":{"rendered":"Running taskcluster tasks locally"},"content":{"rendered":"<h3><i>Work right from your own home!<\/i><\/h3>\n<p>It can be difficult to debug failures in Taskcluster that don&#8217;t happen locally. Interactive tasks are very useful for this, but interactive tasks broke during the last migration &#8212; a relevant bug is <a href=\"https:\/\/bugzilla.mozilla.org\/show_bug.cgi?id=1596632\">bug 1596632<\/a>, which is duped to a just-fixed bug, so maybe it works now?. I recently encountered a situation where I really needed to interactively debug something, so I decided to take the plunge and discover the answer to the question: how can I run tasks locally?<\/p>\n<p>Local tasks provide not only the advantages of interactive tasks, but also allow running against your local checkout. That makes for a much faster edit-run-curse-debug cycle, and opens up possibilities for using this in a lot more situations than the usual last-ditch efforts that interactive try server tasks are usually used for. (Or at least, that&#8217;s how I use them. And mostly don&#8217;t use them.)<\/p>\n<p>I&#8217;m going to walk through the process of setting up and running a taskcluster job in a local container. Note that I have no idea how generally applicable this is. I will give the steps necessary to run the SM(gdb) job, which builds the JS shell and runs some gdb prettyprinter tests against it. I have no idea how far it will get you to running something like mochitests.<\/p>\n<h3>Getting the image<\/h3>\n<p>Taskcluster normally runs Docker images. So the first step is to get your very own copy of the appropriate docker image. There&#8217;s <a href=\"http:\/\/code.v.igoro.us\/posts\/2016\/04\/taskcluster-load-image.html\" rel=\"noopener noreferrer\" target=\"_blank\">a handy blog post by someone who actually knows what he&#8217;s talking about<\/a> that I found well after the fact (of course). But I&#8217;m going to give the exact steps that I used:<\/p>\n<ul>\n<li>Click on the task you&#8217;re trying to replicate in treeherder.\n<li>Open the full log file.\n<li>Search for a line that says something like &#8220;Downloading artifact &#8220;public\/image.tar.zst&#8221; from task ID: VuFo68PeQjCH7k15tSN2Dg.&#8221; near the beginning of the file. Call that ID <code>$IMAGEID<\/code>.\n<li>Run <code>.\/mach taskcluster-load-image --task-id $IMAGEID<\/code> from your Gecko checkout.\n<\/ul>\n<p>and then optionally,<\/p>\n<ul>\n<li>Curse and flail around when something goes wrong with the docker import process, as it always seems to.\n<li>Maybe install docker in the first place. Whoops, forgot to mention that.\n<li>You probably want it to be running as well.\n<\/ul>\n<h3>Getting the image up and running<\/h3>\n<p><code>mach<\/code> will helpfully give you a command to run a shell in the image, something like <\/p>\n<pre>\/usr\/bin\/docker-current run -ti --rm debian7-amd64-build:e2e821aea119e4a264340c22b79324ac804955b605577dd225df5f4f8e98e0cc bash<\/pre>\n<p>. Don&#8217;t do that. It&#8217;s a great command, but it&#8217;s a little overzealous about cleaning up after itself. But grab out that image name: <code>IMAGE=debian7-amd64-build:e2e821aea119e4a264340c22b79324ac804955b605577dd225df5f4f8e98e0cc<\/code><\/p>\n<p>Although for now, I guess it&#8217;s really not bad. Just remove the <code>--rm<\/code> option and give it a try.<\/p>\n<p>If you get a shell to pop up, congratulations! Be happy! If not, try asking someone with a clue or, failing that, ask me. I&#8217;m <code>sfink<\/code> in the <code>#developers<\/code> channel on IRC, or if you&#8217;re reading that after we&#8217;ve spun up our new Matrix overlord, I&#8217;ll probably be moving there. Oh, and if you&#8217;re in the Mozilla secret club, I suppose I won&#8217;t ignore you if you hit me (<code>@sfink<\/code>) up on Slack either.<\/p>\n<p>Anyway, we&#8217;re going need to download some stuff into this image, which means we need a network. Mine didn&#8217;t start with a network. I don&#8217;t know much about Docker, but this got me a network:<\/p>\n<ul>\n<li><code>ifconfig<\/code> to figure out your local IP address, or do it some other way. My IP was 10.0.0.14.\n<li>\n<pre>docker network create -o \"com.docker.network.bridge.host_binding_ipv4\"=\"10.0.0.14\" my-network<\/pre>\n<p>, replacing the &#8220;10.0.0.14&#8221; with your own IP and, if you wish, &#8220;my-network&#8221; with something cooler-sounding. That&#8217;ll spit out some monstrous ID like <code>1793d9caad6d5973922b7a78ae11a2bce6005781ca18c0e253d1c2c5317f5c93<\/code> that you have to read out in Pig Latin in under 5 seconds. Or you can just ignore it.<\/p>\n<li><code>docker ps<\/code> to get the ID of your running container. (Or add <code>-a<\/code> if you&#8217;re going to be running a container you&#8217;ve created already.) Call that $CONTAINER_ID.\n<li><code>docker network connect my-network $CONTAINER_ID<\/code>\n<\/ul>\n<p>Come to think of it, I only did that once with an old container I&#8217;m not longer using, and all of the new containers I&#8217;ve created come up with a functioning network from the get-go. So you can probably ignore all of the above.<\/p>\n<h3>Grafting your source into your container<\/h3>\n<p>Now that you have a container with a network running and everything, it&#8217;s time to throw it out and start over. I did say &#8220;don&#8217;t do that&#8221;, remember?<\/p>\n<p>The next goal is to start up a container with your local source tree bind-mounted. Let&#8217;s call the absolute path to your checkout <code>$SRCDIR<\/code>.<\/p>\n<ul>\n<li>Let&#8217;s expand your container-creating command to something like:\n<pre>docker run -ti -v $SRCDIR:\/builds\/worker\/source:z $IMAGE bash<\/pre>\n<p>[Note 2]<\/p>\n<li>But don&#8217;t run that either. Or at least, don&#8217;t run it if you actually <i>are<\/i> trying to run the gdb task, because it requires some extra privileges in order to do the right ptrace magic.\n<li>Here&#8217;s the actual command I use:\n<pre>docker run -ti -v $SRCDIR:\/builds\/worker\/source:z --cap-add=SYS_PTRACE --security-opt seccomp=unconfined $IMAGE bash<\/pre>\n<\/ul>\n<p>Ignoring the gdb ptrace goop, what that&#8217;s doing is bind-mounting $SRCDIR on your host so that it shows up at <code>\/builds\/worker\/source<\/code> within your container, and additionally does the fixup necessary for selinux to allow you to then access the data from within the container. If you&#8217;re worried about stuff running within the container messing up your source checkout, you could add <code>,ro<\/code> to the volume portion of that command: <\/p>\n<pre>docker run -ti -v $SRCDIR:\/builds\/worker\/source:z,ro --cap-add=SYS_PTRACE --security-opt seccomp=unconfined $IMAGE bash<\/pre>\n<p>. But honestly, I&#8217;ve never tried doing that yet.<\/p>\n<h3>Snarfing taskcluster initialization<\/h3>\n<p>Hopefully, you now have a shell open in a container that is basically identical to what runs in taskcluster. You&#8217;re home free, right?<\/p>\n<p>Not so fast. Taskcluster does some magic setup, I&#8217;m not entirely sure how, to provide an environment with a bunch of important settings that don&#8217;t come with a default shell. I figured out a bunch of stuff you could do manually to replicate this environment. Here&#8217;s a list of steps that I recommend you do <b>not<\/b> take:<\/p>\n<ul>\n<li>Go back to your push on treeherder.\n<li>Click on the <b><code>Task<\/code><\/b> link in the bottom left pane.\n<li>Expand the &#8220;payload&#8221; section.\n<li>Somehow convert the whole &#8220;env&#8221; section to environment variable setting commands. I used to save the whole payload as a JSON file <code>\/tmp\/task.json<\/code>, then run\n<pre>perl -lne 'if (\/\"env\"\/ .. \/^\\s*\\}\/) { print \"export $1='\\''$2'\\''\" if \/\"(.*?)\": \"(.*)\"\/ }' \/tmp\/task.json<\/pre>\n<li>Cut &#038; paste that into the shell running on your container.\n<li>Also cut &#038; paste\n<pre>export TASKCLUSTER_ROOT_URL=https:\/\/firefox-ci-tc.services.mozilla.com<\/pre>\n<p> to prevent it from attempting to access stuff via internal URLs that won&#8217;t work from your desktop.<\/p>\n<li>Now grab the &#8220;command&#8221; key from that payload and stitch it together into a shell command to paste&#8230;\n<ul>\n&#8230;but that&#8217;s way too much work, and is incomplete besides.<\/p>\n<h3>Running the command<\/h3>\n<p>That last step, where you grabbed the command out of the payload? It&#8217;s not going to work. The main reason is that it attempts to do some hg fingerprinting thing that won&#8217;t work when you&#8217;re running outside of the data center. But the automation I created to avoid that also does the environment initialization piece, so I&#8217;ll whack the two little birdies with one rock.<\/p>\n<ul>\n<li>Download <a href=\"https:\/\/raw.githubusercontent.com\/hotsphink\/sfink-tools\/master\/bin\/mk-task-runner\">https:\/\/raw.githubusercontent.com\/hotsphink\/sfink-tools\/master\/bin\/mk-task-runner<\/a> or checkout all of <code>https:\/\/github.com\/hotsphink\/sfink-tools<\/code> and find it in <code>bin\/<\/code>.\n<li>Look at the bottom left pane for the <b><code>Task<\/code><\/b> field of the job you&#8217;re cloning. Copy the magic task ID next to it, something like <code>fpcWJf1hTEinv8F49luc_w<\/code>. Let&#8217;s call that <code>$TASKID<\/code>.\n<li>From within your source checkout, run <code>mk-task-runner $TASKID<\/code>. That&#8217;ll download the task descriptor and grab out the relevant pieces, and generate a simple <code>run-task.sh<\/code>.\n<li>Because this is in your source checkout, it should be visible from within the container. So run <code>source\/run-task.sh<\/code>.\n<\/ul>\n<p>This should run the whole task, and after it&#8217;s done, drop you into another shell with the environment settings preserved in case you want to do some further poking around.<\/p>\n<p>&#8230;except, once again, it probably won&#8217;t work. <\/p>\n<h3><code>run-task<\/code><\/h3>\n<p>Tasks run via a script <code>taskcluster\/scripts\/run-task<\/code>. Which is great, and it <i>almost<\/i> works perfectly for our purposes. Except it tries to do its own checkout of gecko, and it spends a bunch of time downloading stuff and then deletes it at the end. Both of those are not so helpful if you&#8217;re trying to run and rerun against your own checkout.<\/p>\n<p>I have <a href=\"https:\/\/bugzilla.mozilla.org\/show_bug.cgi?id=1605232\" rel=\"noopener noreferrer\" target=\"_blank\">bug 1605232<\/a> open for patches that add options to avoid that, but (1) it hasn&#8217;t landed, (2) it hasn&#8217;t been reviewed, (3) it may not be the direction The Powers That Be want to go, and (4) they might really rather not be using <code>run-task<\/code> for both automation and manual running in the first place. All of which could lead to this solution changing. If I&#8217;m a good person[Note 1], I&#8217;ll come back to this post and update it with the updated information when things change.<\/p>\n<p>In the meantime, you have two main options:<\/p>\n<ol>\n<li>Edit <code>run-task.sh<\/code> to get rid of the <code>--keep<\/code> and <code>--existing-gecko-checkout=...<\/code> options and let it run against a fresh checkout a re-download stuff, or\n<li>Apply the patch in the above bug to your local checkout.\n<\/ol>\n<h3>Aftermath<\/h3>\n<p>I was careful to post this during the holiday season to be sure you wouldn&#8217;t read it, but it looks like you somehow did anyway. If taskclustery people who actually work on this stuff would like to correct my undoubtedly numerous mistakes, I would be most appreciative, so please get in touch. Or if you can tell me where I&#8217;m making it all too hard.<\/p>\n<p>If you use this and it works for you, I&#8217;d be curious to know what you&#8217;re using it for. If you try to use it and it doesn&#8217;t work, I&#8217;d kinda like to know that too (I haven&#8217;t tried any other tasks yet.) If you try to use it and get angry about it not working, or it eats your data, I&#8217;m perfectly okay with you not getting in touch with me.<\/p>\n<hr>\n<p>Footnote 1: I&#8217;m not a good person.<\/p>\n<p>Footnote 2: The documentation says you should really be using <code>--mount<\/code> in place of <code>-v<\/code> aka <code>--volume<\/code>. But my version of docker doesn&#8217;t have <code>--mount<\/code>.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Work right from your own home! It can be difficult to debug failures in Taskcluster that don&#8217;t happen locally. Interactive tasks are very useful for this, but interactive tasks broke during the last migration &#8212; a relevant bug is bug 1596632, which is duped to a just-fixed bug, so maybe it works now?. I recently [&hellip;]<\/p>\n","protected":false},"author":206,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5],"tags":[666,137,448592],"_links":{"self":[{"href":"https:\/\/blog.mozilla.org\/sfink\/wp-json\/wp\/v2\/posts\/378"}],"collection":[{"href":"https:\/\/blog.mozilla.org\/sfink\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.mozilla.org\/sfink\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/sfink\/wp-json\/wp\/v2\/users\/206"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.mozilla.org\/sfink\/wp-json\/wp\/v2\/comments?post=378"}],"version-history":[{"count":0,"href":"https:\/\/blog.mozilla.org\/sfink\/wp-json\/wp\/v2\/posts\/378\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.mozilla.org\/sfink\/wp-json\/wp\/v2\/media?parent=378"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.mozilla.org\/sfink\/wp-json\/wp\/v2\/categories?post=378"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.mozilla.org\/sfink\/wp-json\/wp\/v2\/tags?post=378"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}