Joey Armstrong's blog

makefiles

adirondackfirefly — Thu, 12 Apr 2012 21:10:36 +0000

Threadsafe mkdir made easy

An interesting view outside this morning

adirondackfirefly — Thu, 15 Mar 2012 13:42:07 +0000

A double rainbow: photo #1, photo #2.

makefile landmines

adirondackfirefly — Tue, 06 Mar 2012 15:55:03 +0000

Makefile landmines: avoiding limb loss

[ps] ignore a similar makefile post I made yesterday. Somehow draft+update implied publish and the original content was not yet ready to see the light of day.

When writing makefiles there are a few logic bombs to try and avoid as they can mask failure conditions. When command exit status is ignored or overlooked the site can become a source for random failures or manifest in downstream testing.

A few problem sources to mention are:

Not returning, propagating, testing for or ignoring shell exit status.
Use of line continuation to join multi-line commands within a makefile.
Prefixing commands with a hyphen ‘-’ to ignore errors.

A simple makefile (gotcha.mk) is attached below that will illustrate problem space for items #1 and #2 using the true & false shell commands. True will succeed with a shell exit status of zero (0==$?). False will fail with a non-zero exit status:

% gmake -f gotcha.mk show
makefile target is: show
     /bin/true: $?=0
    /bin/false: $?=1
 
% gmake -f gotcha.mk pass
makefile target is: pass
/bin/true
 
% gmake -f gotcha.mk fail
makefile target is: fail
/bin/false
gmake: *** [fail] Error 1

Line continuation: why they can be a problem

Line continuation is an efficiency that is often used to avoid the overhead of having to spawn a new command shell for each command mentioned in a target rule. Most of the time line continuation will function as expected but when exit status is lost or over-written, success is returned allowing make to return a false-positive.

An easy way to create this condition is by wrappering the logic in an if block. Commands can merrily fail in bulk within the block but final exit status will be set by the shell if/test condition — which always succeeds.

This can be illustrated by the makefile target ‘bad‘. Run the true command, then false and if all is well the target should fail on the false call. Guess what, we have opened a logic hole for bugs to quietly sneak in through:

% gmake -f gotcha.mk bad
makefile target is: bad
if [ ! -e '/non/existent/file' ]; then \
            /bin/true; \
            /bin/false; \
            echo "ASSERT: should not reach this point"; \
        fi
ASSERT: should not reach this point

There are a few workarounds for this problem space. One of the easiest is use of the shell ‘-e’ set/command line arg. When enabled a shell will exit immediately with non-zero status whenever a command fails.

Uncomment this line in the gotcha.mk and the command will fail as expected:
SHELL := $(SHELL) -e

% gmake -f gotcha.mk bad
makefile target is: bad
   [snip]
gmake: *** [bad] Error 1

The -e option or a morale equivalent is supported by bourne, bash, dash, ksh or whatever other shells people may prefer to use.

suggestions: line continuation alternatives

At a minimum for readability consider using GNU make’s .ONESHELL: directive in place of large line continuation blocks: http://www.gnu.org/software/make/manual/html_node/One-Shell.html
Avoid line continuation wherever possible. Make is very good at control flow and processing but not so much as a scripting language. Blocks that span more than ~5 commands or are encapsulated within an if block would be ripe for breaking up into several distinct makefile goals {better yet general targets in a makefile library} or moving the entire block into a shell script that will be invoked by the target.

shell/set -e flags should be enabled everywhere to improve error detection along with options to detect uninitialized variable usage.  Functionality is already enabled by in a few places with use of the $(EXIT_ON_ERROR) make macro defined in config/config.mk.  Listing $(EXIT_ON_ERROR) as the first item of a target rule will be expanded by make into $(SHELL) -e @command_text enabling fail on all command failure handling:
config/config.mk

50
EXIT_ON_ERROR = set -e; # Shell loops continue past errors without this.

config/rules.mk

226
227
228
229
230
check::
  @$(EXIT_ON_ERROR) \
    for f in $(subst .cpp,$(BIN_SUFFIX),$(CPP_UNIT_TESTS)); do \
      XPCOM_DEBUG_BREAK=stack-and-abort $(RUN_TEST_PROGRAM) $(DIST)/bin/$$f; \
    done

Make judicious use of sh -e functionality but do not blindly trust that the option has been enabled for a given makefile target.  There will not always be a visual cue that the flag is active. In this case if the EXIT_ON_ERROR macro were inadvertently cleared or commented out a subset of error detection would be lost.  Best option initially is to manually verify the failure condition then write a negative unit test to back up the assertion long term.

One trivial way to validate logic blocks like these is leverage the try server and modify the makefile to perform a negative test.  Simply inline a call to 'exit ###' as the last statement in a block.  Submit makefile edits that are guaranteed to fail then verify all platforms reported the failure everywhere: in logfiles, email and tbpl web status.

all:
    if [ 1 -eq 1 ]; then\
    /bin/ps \
    && /bin/who \
    && echo "********************************"\
    && echo "Failure forced"\
    && echo "********************************"\
    && exit 99\
    fi
code>


% ${EDITOR} makefile.in
% hg qnew patch.try
% hg qref --message "try: -b do -e -p all -u all -t none"
% hg push -f try

If all goes well failure email will soon be on it's way.  If the makefile fails with status 99 the negative test succeded.  Be careful of errors reported early by ps or who, status could still be ignored by logic later on.  ($?==99) is the only exit status that should be used to assert the negative test passed:

gmake: *** [all] Error 99

Negative testing can be as important as positive testing and will sanity check two important items.  First that the exit/error status was properly detected and make was able to generate and record an error in the logs.  Second, the failure status was able to propagate through the try server and was reported accurately by the web interface and through status email.
If either condition is not met a bug should be opened so the logic can be modified to prevent any future failure conditions from silently sneaking into the tree.  If negative tests are not able to force a failure conditions valid errors that occur while building/processing will not either.
The configure script would also be a place to perform similar -e tests.

     Often when commands cannot be found or failures occur the configure script

     is able to run to completion but underlying failures are lost in the

     shuffle.
 Lastly a plug for testing.  If you will be expending effort to test makefile logic go the extra setup and write a unit test and add a 'check:' target to automate the activity.  Even if the test is a trivial smoke test initially it can be expanded later.  Start with these three conditions, if any fail something has gone horribly wrong and may needlessly waste queue time and build resources:

positive test: target does not exist initially, generate and verify.
negative test: target cannot be created and test fails.
nop test: target exists, minimal overhead should be imposed.  If a target performs work when up to date think of how much time is being wasted by the sum of all targets with this behavior.
     

% gmake -f gotcha.mk check
makefile target is: check
Running positive test
OK
Running negative test
OK
Running negative test [failure forced]
gmake: *** [check] Error 1


Ignoring errors: hyphen prefix for commands
AKA I do not care what type of explosion occurs on this line, ignore status and always return success.

all:
    @echo "Inhibit target rule errors"
    -/bin/false

There are a few legitimate places where this option can be used.

File deletion on a system that does not support rm -f.
Shell/OS command bug that will force returning non-zero status for

command that should succeed (~transient problems).
Aside from cases like these, use of a hyphen in general makefile logic to ignore a set of failure conditions will have an ugly side effect of also ignoring an entire set of unrelated problems.  In the filesystem and OS area, that a target should be aware of and fail on.
Ignoring conditions like say a file being conditionally non-existent when a target is processed because the file will be generated by the target is one example.  Hmmm, this might also be an instance of trying to work around a line continuation problem, testing for file existence and subsequent loss of command exit status within the if block.
In place of using hyphens to code around a set of conditions remove them outright and use an external shell script to do the right thing.  Handle conditions, test for existence and return an overall shell exit status from the actions.  If anything fails always return non-zero.
Samples:
Here is a sampling of makefile code from the tree with problems mentioned above.  Not trying to pick on anything specific, just what can be found with a bit of poking around.


toolkit/locales/l10n.mk

** -cp -r $(JARLOG_DIR)/en-US/*.jar.log $(JARLOG_DIR_AB_CD)

** -$(PERL) -pi.old -e "s/en-US/$(AB_CD)/g" $(JARLOG_DIR_AB_CD)/*.jar.log

** A few conditions these commands will succeed on:

*** Source files do not exist.  When file existence is not mandatory $(if ) conditions and extra target rules can conditionally copy when needed.  As could an external shell script that could bundle several statements and eventually be tested.

*** Destination directory does not exist or filesystem is full.

*** Source file is not readable.

*** Destination file exists but is not writable.

*** $(JARLOG_DIR_AB_CD) does not exist and we attempt to copy files into the directory.

js/src/ctypes/libffi/Makefile.in

** -rm -f src/alpha/ffi.$(OBJEXT)

*** -rm is implied by the $(RM) macro: RM = rm -f

*** Written as: $(RM) src/alpha/ffi.$(OBJEXT)

** -test -z "$(lib_LTLIBRARIES)" || rm -f $(lib_LTLIBRARIES)

** -test -z "$(noinst_LTLIBRARIES)" || rm -f $(noinst_LTLIBRARIES)

*** Avoid spawning extra shells

*** $(RM) will not fail when the macros are empty

*** Written as: $(RM) $(lib_LTLIBRARIES) $(noinst_LTLIBRARIES)

config/rules.mk

* clean clobber realclobber clobber_all:: $(SUBMAKEFILES)

* -$(RM) $(ALL_TRASH)

* -$(RM) -r $(ALL_TRASH_DIRS)

** -$(RM) is redundant of $(RM) unless RM != /bin/rm -f

Sample makefile: gotcha.mk

# -*- makefile -*-
###############################################
## Intent: makefile landmine examples
###############################################
 
ifneq ($(null),$(MAKECMDGOALS))
  $(info makefile target is: $(MAKECMDGOALS))
endif
 
SHELL := $(SHELL) -e
 
all: pass fail bad
 
pass:
        /bin/true
 
fail:
        /bin/false
 
bad:
        if [ ! -e '/non/existent/file' ]; then \
            /bin/true; \
            /bin/false; \
            echo "ASSERT: target $@: should not reach this point"; \
        fi
 
check:
        @echo "Running positive test"
        @/bin/true && echo "OK" || exit 1
        @echo "Running negative test"
        @! /bin/false && echo "OK" || exit 1
        @echo "Running negative test [failure forced]"
        @/bin/false && echo "OK" || exit 1
 
show:
        @if [ 1 ]; then\
            /bin/true;  echo "     /bin/true: \$$?=$$?"; \
            /bin/false; echo "    /bin/false: \$$?=$$?"; \
        fi



Build system tools: make-makefile, file generation
Fri, 17 Feb 2012 18:29:35 +0000
This post will be article #2 in a series about the build-config tool make-makefile.

The series is being written to formally document current tool functionality and

provide a base for future topics related to the container makefile project.

Container structure, functionality and example usage.
Series posts

#2: A cookbook for makefile generation
#1: intro

When makefile generation is needed the tool will perform a few steps.

gmake -f client.mk is invoked. During traversal if gmake detects one

of two conditions the tool will be invoked to generate a Makefile:

    o $(obj)/Makefile does not exist.

    o $(src)/Makefile.in is newer than $(obj)/Makefile.
Target rules used to invoke mm live in config/rules.mk near line 1342 and 1347

    o 1342 Makefile: Makefile.in

    o 1343 @$(PERL) $(AUTOCONF_TOOLS)/make-makefile -t $(topsrcdir) -d $(DEPTH)
make-makefile will be passed -d and -t command line arguments:

    o -t path to root of a development sandbox ~$(TOPSRCDIR)

    o -d relative path –depth from file to hierarchy root directory.

Makefile: $(MOZ_OBJDIR), Makefile.in: $( TOPSRCDIR)
When invoked make-makefile is aware of the directory hierarchy and will expect cwd for the shell to be $(MOZ_OBJDIR) or a subdir of it [1]. config/rules.mk logic will place make-makefile in the target directory for each makefile to be generated [2].
Step 1 – obtain values for depth, topsrcdir and objdir.
Values will be:

    o used to construct directory and file paths.

    o perform text substitution within Makefile.in templates.
Values can be:

    o explicitly passed on the command line.

    o determined from the filesystem based on arguments and/or cwd.

    o extracted from file arguments.
$depth will be set by:

    o Explicitly using the -d command line argument.

    o Assignment of DEPTH=../.. specified within an individual Makefile.

    o Assignment of DEPTH=../.. within any other files passed on the command line

      (is this behavior a bug or feature?).

    o Assignment of DEPTH=../.. contained within a parent directory Makefile [3], [4].
Step 2 – Using cwd and arguments, derive paths to the source template

Makefile.in and generated target file Makefile.
Step 3 – Slurp the source template Makefile.in. Perform value substitutions

on tokens embedded within the template of the form @token@.

browser/branding/official/Makefile.in
=====================================
DEPTH = ../../..
topsrcdir = @topsrcdir@
srcdir = @srcdir@
VPATH = @srcdir@

Some directory/file paths are derived internally by the tool for quick substitution. For any @${unknown}@ tokens make-makefile will delegate expansion with a call to system(“./config.status”) [5] to obtain values, create directories and who knows what else. Shell overhead will be imposed by this step so avoid unnecessary tokens when possible. When only a handful of tokens are in play one optimization to check on is if mm could derive values for the tokens and avoid overhead from calling the config.status script [6].
Step 4 – update or preserve generated Makefile timestamps. If obj/Makefile does not exist create it. If the file exists compare generated content against the original and only update when modified.

footnotes


[1]
The requirement of cwd==$(MOZ_OBJDIR)/ can be removed with use of the –enhanced flag. This option and other new flags will be covered in a future post.

[2]
Enhancement: a response file could be used to support bulk makefile processing and directory creation (by config.status) saving some shell overhead. Bulk processing might also reduce the number of times config.status must be invoked independently.

[3]
Potential bug. DEPTH= searches within multiple files could set $depth incorrectly when it is not explicitly set within a makefile. One ex would be passing browser/…/Makefile and js/src/…/Makefile as command line arguments. Though this error condition exists it is not likely to be triggered as files are processed individually.
  

[4]
Potential bug: checking ../Makefile for ‘DEPTH=’ will fail for makefiles invoked from a parent directory several layers above cwd.

[5]
Enhancement – modify make-makefile to parse and extract fully expanded values from config* to avoid invoking ./config.status for a subset of substitution tokens.

[6]
make-makefile will issue a warning when the ./config.status script will be launched

WARNING: token SET_MAKE not defined

line 2, src: [.....]/js/src/ctypes/libffi/Makefile.in


resources
Tool source

build/autoconf/make-makefile
build/autoconf/make-makefile.pm
build/autoconf/make-makefile.excl

Unit tests

build/autoconf/test/Makefile.in

glob *.tp{l,m} scripts and launch as standalone unit tests.
build/autoconf/test/make-makefile.tpl
build/autoconf/test/makemakefile.tpm

makefiles

config/rules.mk  [target rule]


Pending enhancements

Expand test coverage for make-makefile
Add Response file support.
Rewrite unit tests in python.
Porting make-makefile from perl to python.
minimize shell overhead from config.status use

Future Blog Topics

make-makefile args and enhancements to generalize logic and support container builds.
Container makefiles: scripts & config

containermake.py
template files
non-invasive makefile edits to support container builds
  
Container makefiles: bulk file processing

Makefile.in => Makefile
*.idl => .h, .xpt, and deps .xpt.pp
makefile targets: export, tools, libs, test, *
thread mutex from target modifiers and extra container dependencies

Makefiles and library logic

threadsafe mkdir library function
config/rules.mk modularity – isolating logic based on bulk build types
thread mutex from target modifiers and container dependencies





Build system tools: make-makefile
Thu, 15 Dec 2011 15:51:24 +0000
Build system tools: make-makefile
The build process consists of several steps, one of which is the dynamic generation of makefiles – the creation of recipes that control how and when elements of firefox and friends are generated.  The tool currently used for generation is make-makefile, a utility written way back in the days of Netscape (circa 1999).  Intent for this post and a few more that will follow will be to document the tool and usage.
Program and config

* build/autoconf/make-makefile

* build/autoconf/makemakefile.pm

* build/autoconf/make-makefile.excl
make-makefile will be automatically invoked while building whenever a Makefile does not exist beneath the object directory or a template (Makefile.in) is newer than a previously generated Makefile.
 



NYC – The City that Never Sleeps? Not so much.
Fri, 05 Aug 2011 17:15:14 +0000
On Friday, July 29, 2011, after completing my first work week with the rest of the Release Engineering team in Toronto I left for the airport around 3pm to catch a flight home to Albany, NY. The week went well and everyone had a great time, visited plenty of good restaurants and pubs along with a trip to the new office (under construction). Little did I know what adventure lay ahead on the trip back home. Here is the abridged version of the chain of events.

My Air Canada flight from Toronto to La Guardia was scheduled to depart @ 7:15pm EDT. Upon arrival at the airport the ticket agent mentioned that the flight had been delayed. My connection from NYC to Albany would be missed, but figured the flight could be straightened out on the ground in LaGuardia.
Sent a text to my wife Lisa to let her know what was going on and she told me heavy thunder storms were in the forecast, moving out around 10-11pm.
Not long before we were supposed to board the plane, the flight was delayed a 2^nd time to 9:00pm EDT
A 3^rd gate change delayed departure until 9:40pm EDT.
Finally boarding, the flight was supposed to leave at 9:40pm, but we are still sitting on the runway at 9:48. We spent another 20-30 minutes taxiing and waiting for overhead traffic to clear but at least we are on the plane!
Oh, did I forget to mention one kid played basketball in the waiting area for 2hrs. Dribbiling, lay-ups on glass by the check in desk. What a special treat for everyone in the waiting area…
Landed at La Guardia around 11pm on Friday night to find out that the US Air flight @ 9:29pm was the ONLY flight available and was clearly missed. The earliest flight out to Albany would be sometime on Sunday.

Yes things were not looking good. Called Lisa again to let her know what was going on.

Began calling car rental companies from the airport. Lovely, these multiple choice answers can be used for all of them:


Business is closed after 10pm
All vehicles have been rented
Vendor will not rent vehicles one-way, must be picked up and dropped off at LaGuardia.


So, onto hotels, maybe I could rest for the night and try to pick up an early rental return. Well Deja-vu:


No Vacancy – on a Friday in NYC, go figure…
No shuttle service available between the hotel & airport
No answer at the main desk or through central reservations


Called Lisa back with an update, with her having never driven in NYC let alone La Guardia and the fact that it was getting late with a ~3.5 hour drive looming overhead. Having her drive down for a pickup did not seem like a good idea.
In the meantime, Lisa was trying to find alternate transportation. She checked the Amtrak schedule but both early morning trains on Saturday were sold out. Also friends we know in NYC were out of town so the streak of bad luck continued.
Lisa found the Metro-North Railroad (commuter train) schedule online and suggested I take a cab to Grand Central Station and try to catch a train to Poughkeepsie NY. The drive would be less than 2 hours from home and she could pick me up. We thought worst case scenario, at Grand Central Station I would have a place to stay until the next train, so that was the plan.
My cabbie made really good time from the airport, even with some construction delays. Figures, arrival time at Grand Central was not long after the 1:50am train for Poughkeepsie left the station so I was stuck waiting around until 6:20am.

Yes 11 hours and counting.

Well hello there Murphy how have you been? Last train departs Grand Central at 2am and police begin sweeping vagrants out into the street shortly afterward. FYI, the station closes from 2am to 5:30am daily. Two young teenage girls and a boy were in the same predicament but luckily an officer made arrangements for them to remain in the dispatch office rather than having to fend for themselves.
So at 2:00am I joined the ranks of NYC’s homeless. With my knapsack and suitcase in tow started walking the streets of Midtown Manhattan. I can easily say that nothing heightens your awareness like having “tourist”, “mobility impaired” and “come rob me” stenciled in neon on your person while walking around the city.
Must have walked at least 40 blocks around Park Ave. Lugging a knapsack loaded with books and a laptop around for 3+ hours is loads of fun, I highly recommend it especially when you are bored with nothing to do and tired.
Looked for a 24/7 coffee shop nearby but there was nothing open.
It was really hot and muggy that night (mid 70s), while roaming the streets, kept texting Lisa so she would know I was still breathing.

Depressing, saw at least 20 homeless people sleeping on street grates, benches, building steps and in alleyways. Ages ranged from teenagers to the elderly. Some people were so contorted from being slumped over on a hard bench with relaxed muscles there should have been damage from snapping or dislocation. Not sure how two people were even able to breathe given the position they were in. I did see one very odd group of “nappers”. They appeared to be clean/fairly well dressed with a few pieces of luggage, wondered if they also had the pleasure of being booted from Grand Central.

Around 4:45am managed to find a local grocery store open and was able to get something to drink but didn’t want to hang around. If this is the city that never sleeps I am not convinced!
At 5am the entertainment arrived, bars were closing and it was time for the severely inebriated to meander home. Good thing most of the travelers were couples, leaning into each other and staggering provided enough forward momentum w/o drifting out in to the street or light fixtures very often. All in all this was probably the most amusing point of the night.
After completing my walking tour of NYC, shuffled back to Grand Central in time to board the 6:20am train bound for Poughkeepsie.  Lisa sent a message that she was on her way to pick me up.

It is a little over a 90 minute drive from home but near her parents so she knew the area. Neither one of us were able to get any sleep the night before and were running on pure adrenaline.

My train rolled into the Poughkeepsie train station at about 8:30am and Lisa was waiting for me on the platform. We were both tired but we hopped back into the car , made a quick stop for breakfast at a drive through and headed home.
We got home about 10:40am and crashed at about 11am; don’t think we were conscious again until 6:30pm.

So there you have it, 20 hours of non-stop excitement after an awesome work week in Toronto. In the grand scheme of things this is small potatoes but I thought everyone would appreciate a good laugh; Murphy’s Law certainly took over. This concludes my little NYC adventure. I wonder what my next trip has in store… Can’t wait!
Until next time.



Hello from the Adirondack Mts
Fri, 01 Jul 2011 17:30:20 +0000
Welcome to my inaugural blog post! I’ve read plenty of blogs but never saw myself actually writing one, so this is uncharted territory for me.  It sort of feels like it did last Saturday while helping my father back his boat into Chateaugay Lake, I ran out dock while directing him and plunged into the lake, clothes, Droid and all!
I figured I’d stop in and day “Hi” before I run off for 10 days for our annual camping trip in the Adirondack Mountains. We plan these trips a year in advance and hope Mother Nature is kind to us; the weather in the Adirondacks can be unpredictable. Breaking down camp in the pouring rain may be a fun challenge for some but doing it in a torrential downpour is something I could live without doing again for a very long time.
As John stated in his blog post, I am wading through TONS of “stuff” to familiarize myself with Mozilla’s Makefiles. Thank you John for the Welcome BTW and thanks to those who have been patient and answered my questions so far.
http://oduinn.com/blog/2011/06/06/welcome-joey-armstrong-to-release-engineering/
Photos from the trip will be forthcoming, until next time!

[1]	The requirement of cwd==$(MOZ_OBJDIR)/ can be removed with use of the –enhanced flag. This option and other new flags will be covered in a future post.
[2]	Enhancement: a response file could be used to support bulk makefile processing and directory creation (by config.status) saving some shell overhead. Bulk processing might also reduce the number of times config.status must be invoked independently.
[3]	Potential bug. DEPTH= searches within multiple files could set $depth incorrectly when it is not explicitly set within a makefile. One ex would be passing browser/…/Makefile and js/src/…/Makefile as command line arguments. Though this error condition exists it is not likely to be triggered as files are processed individually.
[4]	Potential bug: checking ../Makefile for ‘DEPTH=’ will fail for makefiles invoked from a parent directory several layers above cwd.
[5]	Enhancement – modify make-makefile to parse and extract fully expanded values from config* to avoid invoking ./config.status for a subset of substitution tokens.
[6]	make-makefile will issue a warning when the ./config.status script will be launched WARNING: token SET_MAKE not defined line 2, src: [.....]/js/src/ctypes/libffi/Makefile.in