Troubleshooting emergent

May 3, 2013
by Wm. Josiah Erikson (wjens)

There’s a problem with emergent and breve – they both segfault regularly, probably because I built them wrong, and Jaime has had to write a little script to detect this and kill/restart them as appropriate (I don’t understand the details of what he’s done, but suffice it to say that I know it’s broken). So I’m working on recompiling it more correctly, on a node where there aren’t stray random libraries in /usr/local, etc

Here are my notes from Trello, where I’m tracking the project. I’m currently recompiling Qt:

emergent won’t compile without GL support in Qt… installing GL libraries on compute-2-1, where I’m compiling… lots of dependencies, total PITA

58 minutes ago
Wm. Josiah Erikson

Qt-everywhere had been built with OpenGL support, since that’s installed on the head node, but it seemed to be broken anyway, we don’t need it, and it’s not installed on the compute nodes, so recompiling without it…

yesterday at 10:53 am
Wm. Josiah Erikson

Compiling on compute-2-1. Also forced 64-bit compile….

yesterday at 10:34 am
Wm. Josiah Erikson

Gonna see if I can compile it on a node.

yesterday at 10:01 am
Wm. Josiah Erikson

Didn’t help. Still crashes in X, too. There are a few more libraries I can install on the nodes to see if it makes any difference, but they’re installed on the head node and it crashes in X there – though we haven’t tried running what he runs through tractor on the head node, which also segfaults. Something else is up…. can’t figure out what. breve segfaults too… what?

May 1 at 8:50 am
Wm. Josiah Erikson

Recompiling with correct lib locations



19 Responses to “Troubleshooting emergent”

  1.   wjens Says:

    Finished compiling, installed. This time it doesn’t crash in the GUI! This bodes well. It does, however, still have the error message:

    qDrawShadePanel: Invalid parameters

    I tried it on the head node, on the node I built on, and then another random node that didn’t have extra libraries installed, and then an AMD node in rack 1, and then compute-4-5. All good! I wonder if this will translate into less idle CPU time and no segfaults? I’ll reboot compute-2-2 so that the kernel log gets cleared, and then I can tell, since there are no timestamps on the segfaults.

  2.   wjens Says:

    Maybe I should recompile breve the same way….

  3.   wjens Says:

    Crap, don’t think it helped with the problem I hoped it would. From compute-2-2, the one I rebooted:

    2467 jdavila 20 0 1791m 341m 43m S 0.0 1.4 0:01.22 emergent
    3624 jdavila 20 0 1791m 339m 43m S 0.0 1.4 0:01.22 emergent
    3155 jdavila 20 0 1791m 339m 43m S 0.0 1.4 0:01.23 emergent
    3479 jdavila 20 0 1791m 339m 43m S 0.0 1.4 0:01.22 emergent

    Four copies of emergent, all using no CPU. There aren’t any segfaults in the kernel log yet though…. I guess that’s good…. we’ll see how long evaluating this generation takes…

  4.   jjdccs Says:

    you’ll probably see emergent starting to move at least a little, but mostly because I restarted the script that checks for emergent “not moving along” on 2-2.

    ‘m still seeing the warning “**WARNING: Unknown type: DataViewer_List”, but I don’t know if that has anything to do with the problem we’re trying to solve. If it doesn’t, next week we should stop my current run and run a bunch of emergent/breve combos on one machine, manually as opposed to via tractor, to see if that makes any difference.

    Are you seeing breve segfaulting on that cpu? Maybe it’s breve causing the problem, somehow, when it calls emergent? In any case, thanks again for your work with this.

  5.   wjens Says:

    emergent no longer segfaults, but breve does, so trying to recompile breve. Currently getting this problem, tried installing and newer mesa-libOSMesa, haven’t finished that yet, but it’ll mean changing it on all the nodes, which would be a PITA. Here’s the current error, maybe I’m barking up the wrong tree, but I’m going home for the weekend now:

    g++ -shared -o bin/osmesaloader.o bin/osmesaloader.cc -L/share/apps/lib -lOSMesa
    g++ -I/share/apps/include -Iinclude/breve -DHAVE_CONFIG_H -I/opt/rocks/include/python2.6 -g -g -O2 -pipe -Wall -fpermissive -c -o bin/breve.o bin/breve.cc
    /usr/bin/ld: /tmp/cc7nnDXo.o: relocation R_X86_64_32 against `.rodata’ can not be used when making a shared object; recompile with -fPIC
    /tmp/cc7nnDXo.o: could not read symbols: Bad value
    collect2: ld returned 1 exit status
    make: [bin/osmesaloader.o] Error 1 (ignored)
    m4 -D_VERSION_=2.7 -D_DATE_=`date +’%m-%d-%y’` README.m4 > README.txt
    m4 -D_VERSION_=2.7 -D_DATE_=`date +’%m-%d-%y’` README_SOURCE.m4 > README_SOURCE.txt
    g++ -I/share/apps/include -Iinclude/breve -DHAVE_CONFIG_H -I/opt/rocks/include/python2.6 -g -g -O2 -pipe -Wall -fpermissive -c -o steve/format.o steve/format.c
    g++ -I/share/apps/include -Iinclude/breve -DHAVE_CONFIG_H -I/opt/rocks/include/python2.6 -g -g -O2 -pipe -Wall -fpermissive -c -o steve/stevelex.o steve/stevelex.c
    bin/breve.cc: In function ‘int main(int, char**)’:
    bin/breve.cc:121: warning: deprecated conversion from string constant to ‘char*’
    bin/breve.cc:151: warning: deprecated conversion from string constant to ‘char*’
    bin/breve.cc:166: warning: invalid conversion from ‘char**’ to ‘const char**’
    bin/breve.cc:166: warning: initializing argument 2 of ‘brEngine* brEngineNewWithArguments(int, const char**)’
    bin/breve.cc: In function ‘void slDemoSpecial(int, int, int)’:
    bin/breve.cc:648: warning: deprecated conversion from string constant to ‘char*’
    bin/breve.cc:652: warning: deprecated conversion from string constant to ‘char*’
    bin/breve.cc:656: warning: deprecated conversion from string constant to ‘char*’
    bin/breve.cc:660: warning: deprecated conversion from string constant to ‘char*’
    bin/breve.cc: In function ‘void slDemoSpecialUp(int, int, int)’:
    bin/breve.cc:673: warning: deprecated conversion from string constant to ‘char*’
    bin/breve.cc:678: warning: deprecated conversion from string constant to ‘char*’
    bin/breve.cc:683: warning: deprecated conversion from string constant to ‘char*’
    bin/breve.cc:688: warning: deprecated conversion from string constant to ‘char*’
    rm lib/libbreve.a
    rm: cannot remove `lib/libbreve.a’: No such file or directory
    make: [lib/libbreve.a] Error 1 (ignored)
    ar -cr lib/libbreve.a simulation/camera.o simulation/drawcommand.o simulation/gldraw.o simulation/image.o simulation/integrate.o simulation/joint.o simulation/lightdetector.o simulation/link.o simulation/movie.o simulation/multibody.o simulation/netsim.o simulation/patch.o simulation/shadowvolume.o simulation/sensor.o simulation/shape.o simulation/springs.o simulation/terrain.o simulation/tiger.o simulation/vclip.o simulation/vclipData.o simulation/volInt.o simulation/world.o simulation/worldObject.o simulation/roam.o simulation/mesh.o simulation/fonts.o simulation/texture.o simulation/skybox.o kernel/breveEval.o kernel/breveObjectAPI.o kernel/callback.o kernel/color.o kernel/engine.o kernel/evalData.o kernel/evalHash.o kernel/evalList.o kernel/frontendAPI.o kernel/interface.o kernel/internal.o kernel/plugins.o kernel/sound.o kernel/url.o kernel/internalFunctions/breveFunctionsCamera.o kernel/internalFunctions/breveFunctionsURL.o kernel/internalFunctions/breveFunctionsControl.o kernel/internalFunctions/breveFunctionsDrawing.o kernel/internalFunctions/breveFunctionsFile.o kernel/internalFunctions/breveFunctionsGraph.o kernel/internalFunctions/breveFunctionsImage.o kernel/internalFunctions/breveFunctionsJoint.o kernel/internalFunctions/breveFunctionsLink.o kernel/internalFunctions/breveFunctionsReal.o kernel/internalFunctions/breveFunctionsMath.o kernel/internalFunctions/breveFunctionsMatrix.o kernel/internalFunctions/breveFunctionsVector.o kernel/internalFunctions/breveFunctionsStatistics.o kernel/internalFunctions/breveFunctionsMenu.o kernel/internalFunctions/breveFunctionsMovie.o kernel/internalFunctions/breveFunctionsMultibody.o kernel/internalFunctions/breveFunctionsNetwork.o kernel/internalFunctions/breveFunctionsNeuralNetwork.o kernel/internalFunctions/breveFunctionsNeural.o kernel/internalFunctions/breveFunctionsObject.o kernel/internalFunctions/breveFunctionsPatches.o kernel/internalFunctions/breveFunctionsPhysics.o kernel/internalFunctions/breveFunctionsPush.o kernel/internalFunctions/breveFunctionsPushCallback.o kernel/internalFunctions/breveFunctionsQGAME.o kernel/internalFunctions/breveFunctionsShape.o kernel/internalFunctions/breveFunctionsSound.o kernel/internalFunctions/breveFunctionsSpring.o kernel/internalFunctions/breveFunctionsStationary.o kernel/internalFunctions/breveFunctionsTerrain.o kernel/internalFunctions/breveFunctionsWorld.o kernel/multithread.o kernel/namespace.o kernel/network.o graph/draw.o graph/graph.o neural/ff.o neural/ffTrain.o neural/Network/neuralNetwork.o neural/Network/neuralNetworkLearningAlgorithm.o steve/format.o steve/stevelex.o steve/steveparse.tab.o steve/breveFunctionsSteveDataObject.o steve/breveFunctionsSteveObject.o steve/breveFunctionsSteveXML.o steve/dataObject.o steve/evalListSort.o steve/evaluation.o steve/expression.o steve/gc.o steve/genetic.o steve/interactive.o steve/object.o steve/steveFrontend.o steve/stringexp.o steve/xml.o steve/pyconvert.o steve/perlconvert.o python/pythonFrontend.o python/breveFunctionsPython.o lisp/lisp.o util/error.o util/hash.o util/list.o util/matrix.o util/memory.o util/nr.o util/quat.o util/stringstream.o util/text.o util/vector.o util/bigMatrix.o
    g++ -o bin/breve -g -O2 bin/breve.o simulation/camera.o simulation/drawcommand.o simulation/gldraw.o simulation/image.o simulation/integrate.o simulation/joint.o simulation/lightdetector.o simulation/link.o simulation/movie.o simulation/multibody.o simulation/netsim.o simulation/patch.o simulation/shadowvolume.o simulation/sensor.o simulation/shape.o simulation/springs.o simulation/terrain.o simulation/tiger.o simulation/vclip.o simulation/vclipData.o simulation/volInt.o simulation/world.o simulation/worldObject.o simulation/roam.o simulation/mesh.o simulation/fonts.o simulation/texture.o simulation/skybox.o kernel/breveEval.o kernel/breveObjectAPI.o kernel/callback.o kernel/color.o kernel/engine.o kernel/evalData.o kernel/evalHash.o kernel/evalList.o kernel/frontendAPI.o kernel/interface.o kernel/internal.o kernel/plugins.o kernel/sound.o kernel/url.o kernel/internalFunctions/breveFunctionsCamera.o kernel/internalFunctions/breveFunctionsURL.o kernel/internalFunctions/breveFunctionsControl.o kernel/internalFunctions/breveFunctionsDrawing.o kernel/internalFunctions/breveFunctionsFile.o kernel/internalFunctions/breveFunctionsGraph.o kernel/internalFunctions/breveFunctionsImage.o kernel/internalFunctions/breveFunctionsJoint.o kernel/internalFunctions/breveFunctionsLink.o kernel/internalFunctions/breveFunctionsReal.o kernel/internalFunctions/breveFunctionsMath.o kernel/internalFunctions/breveFunctionsMatrix.o kernel/internalFunctions/breveFunctionsVector.o kernel/internalFunctions/breveFunctionsStatistics.o kernel/internalFunctions/breveFunctionsMenu.o kernel/internalFunctions/breveFunctionsMovie.o kernel/internalFunctions/breveFunctionsMultibody.o kernel/internalFunctions/breveFunctionsNetwork.o kernel/internalFunctions/breveFunctionsNeuralNetwork.o kernel/internalFunctions/breveFunctionsNeural.o kernel/internalFunctions/breveFunctionsObject.o kernel/internalFunctions/breveFunctionsPatches.o kernel/internalFunctions/breveFunctionsPhysics.o kernel/internalFunctions/breveFunctionsPush.o kernel/internalFunctions/breveFunctionsPushCallback.o kernel/internalFunctions/breveFunctionsQGAME.o kernel/internalFunctions/breveFunctionsShape.o kernel/internalFunctions/breveFunctionsSound.o kernel/internalFunctions/breveFunctionsSpring.o kernel/internalFunctions/breveFunctionsStationary.o kernel/internalFunctions/breveFunctionsTerrain.o kernel/internalFunctions/breveFunctionsWorld.o kernel/multithread.o kernel/namespace.o kernel/network.o graph/draw.o graph/graph.o neural/ff.o neural/ffTrain.o neural/Network/neuralNetwork.o neural/Network/neuralNetworkLearningAlgorithm.o steve/format.o steve/stevelex.o steve/steveparse.tab.o steve/breveFunctionsSteveDataObject.o steve/breveFunctionsSteveObject.o steve/breveFunctionsSteveXML.o steve/dataObject.o steve/evalListSort.o steve/evaluation.o steve/expression.o steve/gc.o steve/genetic.o steve/interactive.o steve/object.o steve/steveFrontend.o steve/stringexp.o steve/xml.o steve/pyconvert.o steve/perlconvert.o python/pythonFrontend.o python/breveFunctionsPython.o lisp/lisp.o util/error.o util/hash.o util/list.o util/matrix.o util/memory.o util/nr.o util/quat.o util/stringstream.o util/text.o util/vector.o util/bigMatrix.o -L/share/apps/lib -Wl,-E -lode -lgsl -lpng -lgslcblas -lfreetype -lreadline -lhistory -ltermcap -ljpeg -lexpat -lz -lm -lglut -lGLU -lGL -lX11 -lXi -lXmu
    ranlib lib/libbreve.a
    bin/breve.o: In function `main’:
    /home/josiah/build/breve_2.7_source/bin/breve.cc:160: undefined reference to `slFormatText(char*)’
    steve/steveparse.tab.o: In function `yyparse()’:
    /home/josiah/build/breve_2.7_source/steve/steveparse.tab.c:3921: undefined reference to `yyerror(char*)’
    /home/josiah/build/breve_2.7_source/steve/steveparse.tab.c:1979: undefined reference to `yylex()’
    steve/steveparse.tab.o: In function `yyparse()’:
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:748: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:748: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:751: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:751: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:720: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:720: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:708: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:708: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:711: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:711: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:714: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:714: undefined reference to `yyfile’
    steve/steveparse.tab.o: In function `stExp’:
    /home/josiah/build/breve_2.7_source/include/breve/expression.h:223: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/include/breve/expression.h:224: undefined reference to `yyfile’
    steve/steveparse.tab.o: In function `yyparse()’:
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:745: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:745: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:732: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:732: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:736: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:736: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:689: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:689: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:699: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:699: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:717: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:717: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:834: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:834: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:835: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:835: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:836: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:836: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:838: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:838: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:754: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:754: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:818: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:818: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:825: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:825: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:826: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:826: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:827: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:827: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:829: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:829: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:784: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:784: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:775: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:775: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:846: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:846: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:846: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:846: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:847: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:847: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:849: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:849: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:852: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:852: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:843: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:843: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:809: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:809: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:811: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:811: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:813: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:813: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:857: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:857: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:858: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:858: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:862: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:862: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:862: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:862: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:862: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:862: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:998: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:998: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1014: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1014: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:920: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:920: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1074: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1074: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1078: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1078: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1093: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1093: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:891: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:891: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:898: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:898: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:506: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:506: undefined reference to `yyfile’
    steve/steveparse.tab.o: In function `stExp’:
    /home/josiah/build/breve_2.7_source/include/breve/expression.h:223: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/include/breve/expression.h:224: undefined reference to `yyfile’
    steve/steveparse.tab.o: In function `yyparse()’:
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1051: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1051: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1051: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1051: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1056: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1056: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1056: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1056: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1194: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1194: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1195: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1195: undefined reference to `yyfile’
    steve/steveparse.tab.o: In function `stExp’:
    /home/josiah/build/breve_2.7_source/include/breve/expression.h:223: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/include/breve/expression.h:224: undefined reference to `yyfile’
    steve/steveparse.tab.o: In function `yyparse()’:
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:871: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:871: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:872: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:872: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:874: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:874: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:874: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:874: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:865: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:865: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:908: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:908: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1026: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1026: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:937: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:937: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1116: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1116: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:905: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:905: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:956: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:956: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:959: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:959: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:959: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:959: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:914: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:914: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:917: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:917: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:911: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:911: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1022: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1022: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:930: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:930: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1106: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1106: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:953: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:953: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1061: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1061: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:943: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:943: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:946: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:946: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1155: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1155: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1036: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1036: undefined reference to `yyfile’
    steve/steveparse.tab.o: In function `stExp’:
    /home/josiah/build/breve_2.7_source/include/breve/expression.h:223: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/include/breve/expression.h:224: undefined reference to `yyfile’
    steve/steveparse.tab.o: In function `yyparse()’:
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:927: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:927: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:940: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:940: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:952: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:952: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1136: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1136: undefined reference to `yyfile’
    steve/steveparse.tab.o: In function `stExp’:
    /home/josiah/build/breve_2.7_source/include/breve/expression.h:223: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/include/breve/expression.h:224: undefined reference to `yyfile’
    steve/steveparse.tab.o: In function `yyparse()’:
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1020: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1020: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1024: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1024: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1029: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1029: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1142: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1142: undefined reference to `yyfile’
    steve/steveparse.tab.o: In function `yyparse()’:
    /home/josiah/build/breve_2.7_source/steve/steveparse.tab.c:4064: undefined reference to `yyerror(char*)’
    steve/steveparse.tab.o: In function `yyparse()’:
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:981: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:981: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:981: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:981: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:993: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:993: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1008: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:1008: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:969: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:969: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveparse.y:442: undefined reference to `yyfile’
    steve/interactive.o: In function `stRunSingleStatement(stSteveData*, brEngine*, char*)’:
    /home/josiah/build/breve_2.7_source/steve/interactive.cc:41: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/interactive.cc:45: undefined reference to `stSetParseData(stSteveData*, char const*, int)’
    steve/steveFrontend.o: In function `stParseError(brEngine*, int, char*, …)’:
    /home/josiah/build/breve_2.7_source/steve/steveFrontend.cc:677: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveFrontend.cc:677: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveFrontend.cc:661: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveFrontend.cc:662: undefined reference to `lineno’
    steve/steveFrontend.o: In function `stPreprocess(stSteveData*, brEngine*, char const*, char const*)’:
    /home/josiah/build/breve_2.7_source/steve/steveFrontend.cc:480: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveFrontend.cc:481: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveFrontend.cc:552: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveFrontend.cc:553: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveFrontend.cc:493: undefined reference to `lineno’
    steve/steveFrontend.o: In function `stParseBuffer(stSteveData*, brEngine*, char const*, char const*)’:
    /home/josiah/build/breve_2.7_source/steve/steveFrontend.cc:433: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveFrontend.cc:435: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveFrontend.cc:442: undefined reference to `yyfile’
    /home/josiah/build/breve_2.7_source/steve/steveFrontend.cc:444: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveFrontend.cc:446: undefined reference to `stSetParseData(stSteveData*, char const*, int)’
    steve/steveFrontend.o: In function `stLoadFiles(stSteveData*, brEngine*, char const*, char const*)’:
    /home/josiah/build/breve_2.7_source/steve/steveFrontend.cc:314: undefined reference to `lineno’
    /home/josiah/build/breve_2.7_source/steve/steveFrontend.cc:316: undefined reference to `yyfile’
    python/pythonFrontend.o: In function `brPythonSetArgv(brEngine*, _object*)’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1141: undefined reference to `PyList_New’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1144: undefined reference to `PyString_FromString’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1146: undefined reference to `PyList_Append’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1151: undefined reference to `PyObject_SetAttrString’
    python/pythonFrontend.o: In function `brPythonSubVectors(_object*, _object*)’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:466: undefined reference to `PyArg_ParseTuple’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:471: undefined reference to `PyObject_CallObject’
    python/pythonFrontend.o: In function `brPythonAddVectors(_object*, _object*)’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:447: undefined reference to `PyArg_ParseTuple’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:452: undefined reference to `PyObject_CallObject’
    python/pythonFrontend.o: In function `brPythonScaleVector(_object*, _object*)’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:429: undefined reference to `PyArg_ParseTuple’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:433: undefined reference to `PyObject_CallObject’
    python/pythonFrontend.o: In function `brPythonSetVector(_object*, _object*)’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:411: undefined reference to `PyArg_ParseTuple’
    python/pythonFrontend.o: In function `brPythonFindInternalFunction(_object*, _object*)’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:593: undefined reference to `PyArg_ParseTuple’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:596: undefined reference to `PyCObject_AsVoidPtr’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:608: undefined reference to `PyCObject_FromVoidPtr’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:604: undefined reference to `PyExc_RuntimeError’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:604: undefined reference to `PyErr_SetString’
    python/pythonFrontend.o: In function `brPythonVectorLength(_object*, _object*)’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:398: undefined reference to `PyArg_ParseTuple’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:403: undefined reference to `PyFloat_FromDouble’
    python/pythonFrontend.o: In function `brPythonCatchOutput(_object*, _object*)’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:358: undefined reference to `PyArg_ParseTuple’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:363: undefined reference to `_Py_NoneStruct’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:363: undefined reference to `_Py_NoneStruct’
    python/pythonFrontend.o: In function `brPythonFindBridgeMethod(_object*, _object*)’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:722: undefined reference to `PyArg_ParseTuple’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:725: undefined reference to `PyObject_GetAttrString’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:740: undefined reference to `PyCObject_AsVoidPtr’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:754: undefined reference to `PyCObject_FromVoidPtr’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:729: undefined reference to `PyExc_RuntimeError’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:729: undefined reference to `PyErr_SetString’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:758: undefined reference to `_Py_NoneStruct’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:758: undefined reference to `_Py_NoneStruct’
    python/pythonFrontend.o: In function `brPythonLoad(brEngine*, void*, char const*, char const*)’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1071: undefined reference to `PyRun_SimpleStringFlags’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1075: undefined reference to `PyRun_SimpleStringFlags’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1083: undefined reference to `PyRun_SimpleFileExFlags’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1097: undefined reference to `PyRun_SimpleStringFlags’
    python/pythonFrontend.o: In function `PyObject_GetAttrStringSafe’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:85: undefined reference to `PyObject_GetAttrString’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:87: undefined reference to `PyErr_Occurred’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:88: undefined reference to `PyErr_Clear’
    python/pythonFrontend.o: In function `brPythonDecodeFromString(brEngine*, char*)’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1193: undefined reference to `PyTuple_New’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1195: undefined reference to `PyString_FromString’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1195: undefined reference to `PyTuple_SetItem’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1197: undefined reference to `PyObject_Call’
    python/pythonFrontend.o: In function `PyObject_GetAttrStringSafe’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:85: undefined reference to `PyObject_GetAttrString’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:87: undefined reference to `PyErr_Occurred’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:88: undefined reference to `PyErr_Clear’
    python/pythonFrontend.o: In function `brPythonEncodeToString(brEngine*, void*)’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1164: undefined reference to `PyTuple_New’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1168: undefined reference to `PyTuple_SetItem’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1170: undefined reference to `PyObject_Call’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1180: undefined reference to `PyString_AsString’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1176: undefined reference to `PyErr_Print’
    python/pythonFrontend.o: In function `brPythonInit(brEngine*)’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1235: undefined reference to `Py_Initialize’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1238: undefined reference to `Py_InitModule4_64′
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1240: undefined reference to `PyCObject_FromVoidPtr’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1241: undefined reference to `PyObject_SetAttrString’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1244: undefined reference to `PyCObject_FromVoidPtr’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1245: undefined reference to `PyObject_SetAttrString’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1250: undefined reference to `PyRun_SimpleStringFlags’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1254: undefined reference to `PyRun_SimpleStringFlags’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1257: undefined reference to `PyImport_ImportModule’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1258: undefined reference to `PyImport_ImportModule’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1268: undefined reference to `PyImport_ImportModule’
    python/pythonFrontend.o: In function `PyObject_GetAttrStringSafe’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:85: undefined reference to `PyObject_GetAttrString’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:87: undefined reference to `PyErr_Occurred’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:88: undefined reference to `PyErr_Clear’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:85: undefined reference to `PyObject_GetAttrString’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:87: undefined reference to `PyErr_Occurred’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:88: undefined reference to `PyErr_Clear’
    python/pythonFrontend.o: In function `brPythonInit(brEngine*)’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1261: undefined reference to `PyErr_Print’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1264: undefined reference to `Py_Finalize’
    python/pythonFrontend.o: In function `PyObject_GetAttrStringSafe’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:85: undefined reference to `PyObject_GetAttrString’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:87: undefined reference to `PyErr_Occurred’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:88: undefined reference to `PyErr_Clear’
    python/pythonFrontend.o: In function `brPythonFindMethod(void*, char const*, unsigned char*, int)’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:876: undefined reference to `PyCallable_Check’
    python/pythonFrontend.o: In function `PyObject_GetAttrStringSafe’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:85: undefined reference to `PyObject_GetAttrString’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:87: undefined reference to `PyErr_Occurred’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:88: undefined reference to `PyErr_Clear’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:85: undefined reference to `PyObject_GetAttrString’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:87: undefined reference to `PyErr_Occurred’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:88: undefined reference to `PyErr_Clear’
    python/pythonFrontend.o: In function `brPythonInstantiate(brEngine*, brObject*, brEval const**, int)’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:954: undefined reference to `PyTuple_New’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:955: undefined reference to `PyObject_CallObject’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:959: undefined reference to `PyErr_Occurred’
    python/pythonFrontend.o: In function `PyObject_GetAttrStringSafe’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:85: undefined reference to `PyObject_GetAttrString’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:87: undefined reference to `PyErr_Occurred’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:88: undefined reference to `PyErr_Clear’
    python/pythonFrontend.o: In function `brPythonInstantiate(brEngine*, brObject*, brEval const**, int)’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:980: undefined reference to `PyCObject_AsVoidPtr’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:960: undefined reference to `PyErr_Print’
    python/pythonFrontend.o: In function `brPythonRemoveInstance(_object*, _object*)’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:567: undefined reference to `PyArg_ParseTuple’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:579: undefined reference to `_Py_NoneStruct’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:579: undefined reference to `_Py_NoneStruct’
    python/pythonFrontend.o: In function `PyObject_GetAttrStringSafe’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:85: undefined reference to `PyObject_GetAttrString’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:87: undefined reference to `PyErr_Occurred’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:88: undefined reference to `PyErr_Clear’
    python/pythonFrontend.o: In function `brPythonRemoveInstance(_object*, _object*)’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:572: undefined reference to `PyCObject_AsVoidPtr’
    python/pythonFrontend.o: In function `brPythonAddInstance(_object*, _object*)’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:489: undefined reference to `PyArg_ParseTuple’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:497: undefined reference to `PyExc_RuntimeError’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:497: undefined reference to `PyErr_SetString’
    python/pythonFrontend.o: In function `PyObject_GetAttrStringSafe’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:85: undefined reference to `PyObject_GetAttrString’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:87: undefined reference to `PyErr_Occurred’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:88: undefined reference to `PyErr_Clear’
    python/pythonFrontend.o: In function `brPythonAddInstance(_object*, _object*)’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:501: undefined reference to `PyCObject_AsVoidPtr’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:510: undefined reference to `PyString_AsString’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:516: undefined reference to `PyCObject_AsVoidPtr’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:527: undefined reference to `PyObject_GetAttrString’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:537: undefined reference to `PyCObject_FromVoidPtr’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:538: undefined reference to `PyObject_SetAttrString’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:551: undefined reference to `PyCObject_FromVoidPtr’
    python/pythonFrontend.o: In function `PyObject_GetAttrStringSafe’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:85: undefined reference to `PyObject_GetAttrString’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:87: undefined reference to `PyErr_Occurred’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:88: undefined reference to `PyErr_Clear’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:85: undefined reference to `PyObject_GetAttrString’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:87: undefined reference to `PyErr_Occurred’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:88: undefined reference to `PyErr_Clear’
    python/pythonFrontend.o: In function `brPythonAddInstance(_object*, _object*)’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:554: undefined reference to `_Py_NoneStruct’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:554: undefined reference to `_Py_NoneStruct’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:534: undefined reference to `PyCObject_AsVoidPtr’
    python/pythonFrontend.o: In function `brPythonSetController(_object*, _object*)’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:374: undefined reference to `PyArg_ParseTuple’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:379: undefined reference to `PyCObject_AsVoidPtr’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:380: undefined reference to `PyCObject_AsVoidPtr’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:390: undefined reference to `_Py_NoneStruct’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:390: undefined reference to `_Py_NoneStruct’
    python/pythonFrontend.o: In function `PyObject_GetAttrStringSafe’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:85: undefined reference to `PyObject_GetAttrString’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:87: undefined reference to `PyErr_Occurred’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:88: undefined reference to `PyErr_Clear’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:85: undefined reference to `PyObject_GetAttrString’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:87: undefined reference to `PyErr_Occurred’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:88: undefined reference to `PyErr_Clear’
    python/pythonFrontend.o: In function `brPythonSetController(_object*, _object*)’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:386: undefined reference to `PyExc_RuntimeError’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:386: undefined reference to `PyErr_SetString’
    python/pythonFrontend.o: In function `brPythonCallMethod(void*, void*, brEval const**, brEval*)’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1006: undefined reference to `PyTuple_New’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1011: undefined reference to `PyTuple_SetItem’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1022: undefined reference to `PyTuple_SetItem’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1025: undefined reference to `PyObject_Call’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1030: undefined reference to `PyErr_Print’
    python/pythonFrontend.o: In function `brPythonCallBridgeMethod(_object*, _object*)’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:776: undefined reference to `PyArg_ParseTuple’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:778: undefined reference to `PyObject_GetAttrString’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:786: undefined reference to `PyCObject_AsVoidPtr’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:790: undefined reference to `PyCObject_AsVoidPtr’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:782: undefined reference to `PyExc_RuntimeError’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:782: undefined reference to `PyErr_SetString’
    python/pythonFrontend.o: In function `brPythonCallInternalFunction(_object*, _object*)’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:624: undefined reference to `PyTuple_Type’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:624: undefined reference to `PyCObject_Type’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:624: undefined reference to `PyArg_ParseTuple’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:627: undefined reference to `PyCObject_AsVoidPtr’
    python/pythonFrontend.o: In function `PyObject_GetAttrStringSafe’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:85: undefined reference to `PyObject_GetAttrString’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:87: undefined reference to `PyErr_Occurred’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:88: undefined reference to `PyErr_Clear’
    python/pythonFrontend.o: In function `brPythonCallInternalFunction(_object*, _object*)’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:648: undefined reference to `_Py_NoneStruct’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:654: undefined reference to `PyCObject_AsVoidPtr’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:660: undefined reference to `_Py_NoneStruct’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:660: undefined reference to `_Py_NoneStruct’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:690: undefined reference to `PyExc_RuntimeError’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:690: undefined reference to `PyErr_SetString’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:637: undefined reference to `PyExc_RuntimeError’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:637: undefined reference to `PyErr_SetString’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:671: undefined reference to `PyExc_RuntimeError’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:671: undefined reference to `PyErr_SetString’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:644: undefined reference to `PyExc_RuntimeError’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:644: undefined reference to `PyErr_SetString’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:700: undefined reference to `PyExc_RuntimeError’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:700: undefined reference to `PyErr_SetString’
    python/pythonFrontend.o: In function `PyObject_GetAttrStringSafe’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:85: undefined reference to `PyObject_GetAttrString’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:87: undefined reference to `PyErr_Occurred’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:88: undefined reference to `PyErr_Clear’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:85: undefined reference to `PyObject_GetAttrString’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:87: undefined reference to `PyErr_Occurred’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:88: undefined reference to `PyErr_Clear’
    python/pythonFrontend.o: In function `brPythonDestroy(void*)’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1131: undefined reference to `Py_Finalize’
    python/pythonFrontend.o: In function `brPythonIsSubclass(brObjectType*, void*, void*)’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:1046: undefined reference to `PyObject_IsSubclass’
    python/pythonFrontend.o: In function `brPythonTypeFromEval(brEval const*, _object*)’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:282: undefined reference to `_Py_NoneStruct’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:282: undefined reference to `_Py_NoneStruct’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:224: undefined reference to `PyInt_FromLong’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:279: undefined reference to `PyCObject_FromVoidPtr’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:228: undefined reference to `PyFloat_FromDouble’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:291: undefined reference to `PyObject_CallObject’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:304: undefined reference to `PyTuple_New’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:306: undefined reference to `PyFloat_FromDouble’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:307: undefined reference to `PyFloat_FromDouble’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:308: undefined reference to `PyFloat_FromDouble’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:309: undefined reference to `PyFloat_FromDouble’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:310: undefined reference to `PyFloat_FromDouble’
    python/pythonFrontend.o:/home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:311: more undefined references to `PyFloat_FromDouble’ follow
    python/pythonFrontend.o: In function `brPythonTypeFromEval(brEval const*, _object*)’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:316: undefined reference to `PyObject_Call’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:232: undefined reference to `PyString_FromString’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:254: undefined reference to `PyTuple_New’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:255: undefined reference to `PyObject_Call’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:266: undefined reference to `PyCObject_FromVoidPtr’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:267: undefined reference to `PyObject_SetAttrString’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:327: undefined reference to `PyList_New’
    python/pythonFrontend.o: In function `PyObject_GetAttrStringSafe’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:85: undefined reference to `PyObject_GetAttrString’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:87: undefined reference to `PyErr_Occurred’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:88: undefined reference to `PyErr_Clear’
    python/pythonFrontend.o: In function `brPythonTypeFromEval(brEval const*, _object*)’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:260: undefined reference to `PyErr_Print’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:262: undefined reference to `_Py_NoneStruct’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:262: undefined reference to `_Py_NoneStruct’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:241: undefined reference to `_Py_NoneStruct’
    python/pythonFrontend.o: In function `PyNumber_AS_DOUBLE(_object*)’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:67: undefined reference to `PyFloat_Type’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:67: undefined reference to `PyFloat_Type’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:67: undefined reference to `PyType_IsSubtype’
    python/pythonFrontend.o: In function `brPythonTypeToEval(_object*, brEval*)’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:109: undefined reference to `_Py_NoneStruct’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:117: undefined reference to `PyFloat_Type’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:117: undefined reference to `PyFloat_Type’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:117: undefined reference to `PyType_IsSubtype’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:127: undefined reference to `PyCObject_Type’
    python/pythonFrontend.o: In function `PyObject_GetAttrStringSafe’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:85: undefined reference to `PyObject_GetAttrString’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:87: undefined reference to `PyErr_Occurred’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:88: undefined reference to `PyErr_Clear’
    python/pythonFrontend.o: In function `brPythonTypeToEval(_object*, brEval*)’:
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:145: undefined reference to `_Py_NoneStruct’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:148: undefined reference to `PyCObject_AsVoidPtr’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:181: undefined reference to `PySequence_Check’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:185: undefined reference to `PySequence_Size’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:190: undefined reference to `PySequence_GetItem’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:129: undefined reference to `PyCObject_AsVoidPtr’
    /home/josiah/build/breve_2.7_source/python/pythonFrontend.cc:138: undefined reference to `PyErr_Print’
    python/breveFunctionsPython.o: In function `brPythonLoadFile(brEval*, brEval*, brInstance*)’:
    /home/josiah/build/breve_2.7_source/python/breveFunctionsPython.cc:38: undefined reference to `PyRun_SimpleStringFlags’
    collect2: ld returned 1 exit status
    make: *** [bin/breve] Error 1

  6.   wjens Says:

    Crap. I was wrong. Emergent still segfaults too. Not sure what to do next:

    emergent[5551]: segfault at 8 ip 000000377a279760 sp 00007fff0a542e50 error 6 in libstdc++.so.6.0.13[377a200000+e8000]
    breve_ex[2650]: segfault at 61684334 ip 00000000084e9d3f sp 00000000ff90fb30 error 4 in breve_ex[8048000+654000]
    breve_ex[2148]: segfault at 61684334 ip 00000000084e9d3f sp 00000000ffdf3720 error 4 in breve_ex[8048000+654000]
    breve_ex[2986]: segfault at 61684334 ip 00000000084e9d3f sp 00000000ff89f480 error 4 in breve_ex[8048000+654000]
    breve_ex[2417]: segfault at 61684334 ip 00000000084e9d3f sp 00000000ffc361f0 error 4 in breve_ex[8048000+654000]
    emergent[8603]: segfault at 8 ip 000000377a279760 sp 00007fff6a693680 error 6 in libstdc++.so.6.0.13[377a200000+e8000]
    breve_ex[25839]: segfault at 61684334 ip 00000000084e9d3f sp 00000000fff01240 error 4 in breve_ex[8048000+654000]
    breve_ex[25606]: segfault at 61684334 ip 00000000084e9d3f sp 00000000ffec7bc0 error 4 in breve_ex[8048000+654000]
    breve_ex[25317]: segfault at 61684334 ip 00000000084e9d3f sp 00000000ffbf3e60 error 4 in breve_ex[8048000+654000]
    breve_ex[26090]: segfault at 61684334 ip 00000000084e9d3f sp 00000000ffd3f0d0 error 4 in breve_ex[8048000+654000]
    emergent[18630]: segfault at ffffffff12f8cce8 ip 0000003776e77771 sp 00007fff25110370 error 4 in libc-2.12.so[3776e00000+186000]
    breve_ex[16638]: segfault at 61684334 ip 00000000084e9d3f sp 00000000ffe34ef0 error 4 in breve_ex[8048000+654000]
    breve_ex[17072]: segfault at 61684334 ip 00000000084e9d3f sp 00000000ff94e8d0 error 4 in breve_ex[8048000+654000]
    breve_ex[16804]: segfault at 61684334 ip 00000000084e9d3f sp 00000000ffc43560 error 4 in breve_ex[8048000+654000]
    emergent[14187]: segfault at 8 ip 000000377a279760 sp 00007fff1f095dd0 error 6 in libstdc++.so.6.0.13[377a200000+e8000]
    breve_ex[17521]: segfault at 61684334 ip 00000000084e9d3f sp 00000000ff9abd40 error 4 in breve_ex[8048000+654000]
    emergent[22974]: segfault at 8 ip 000000377a279760 sp 00007fff0b405990 error 6 in libstdc++.so.6.0.13[377a200000+e8000]
    breve_ex[16364]: segfault at 61684334 ip 00000000084e9d3f sp 00000000ff9d42f0 error 4 in breve_ex[8048000+654000]
    breve_ex[17298]: segfault at 61684334 ip 00000000084e9d3f sp 00000000fff483e0 error 4 in breve_ex[8048000+654000]
    breve_ex[17722]: segfault at 61684334 ip 00000000084e9d3f sp 00000000ff8d7d70 error 4 in breve_ex[8048000+654000]
    breve_ex[16817]: segfault at 61684334 ip 00000000084e9d3f sp 00000000ffa71f80 error 4 in breve_ex[8048000+654000]
    emergent[16923]: segfault at 8 ip 000000377a279760 sp 00007fff8a19f310 error 6 in libstdc++.so.6.0.13[377a200000+e8000]
    breve_ex[9160]: segfault at 61684334 ip 00000000084e9d3f sp 00000000ffbe7fc0 error 4 in breve_ex[8048000+654000]
    breve_ex[9586]: segfault at 61684334 ip 00000000084e9d3f sp 00000000ffeb1240 error 4 in breve_ex[8048000+654000]

  7.   wjens Says:

    I find this interesting:

    top – 10:34:36 up 27 days, 4 min, 1 user, load average: 1.71, 1.73, 1.68
    Tasks: 168 total, 3 running, 164 sleeping, 0 stopped, 1 zombie
    Cpu(s): 0.1%us, 7.7%sy, 0.0%ni, 86.0%id, 0.0%wa, 1.7%hi, 4.4%si, 0.0%st
    Mem: 8158536k total, 4164016k used, 3994520k free, 184636k buffers
    Swap: 5116692k total, 15536k used, 5101156k free, 2826284k cached

    PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
    13 root 20 0 0 0 0 S 41.1 0.0 978:58.94 ksoftirqd/2
    1263 root 20 0 0 0 0 R 15.6 0.0 183:40.70 rpciod/1
    17 root 20 0 0 0 0 S 11.6 0.0 1238:51 ksoftirqd/3
    2 root 20 0 0 0 0 S 8.3 0.0 96:22.57 kthreadd
    60 root 39 19 0 0 0 S 0.3 0.0 18:29.79 khugepaged
    1160 root 20 0 0 0 0 S 0.3 0.0 11:27.29 kondemand/1
    1162 root 20 0 0 0 0 S 0.3 0.0 6:31.21 kondemand/3
    1460 root 20 0 192m 1216 716 S 0.3 0.0 12:01.32 snmpd
    24766 josiah 20 0 15024 1240 896 R 0.3 0.0 0:00.12 top
    This means, maybe, that the CPU is using up tons of time doing something with interrupt requests… I’ll have to look into this further…. I wonder if it would happen if we only ran one copy of emergent per node…. easy enough to check that, I guess!

  8.   wjens Says:

    I think this might indicate an unusually high number of rescheduling interrupts:

    [josiah@compute-1-11 ~]$ cat /proc/interrupts
    CPU0 CPU1 CPU2 CPU3
    0: 132 0 0 1 IO-APIC-edge timer
    1: 0 0 0 2 IO-APIC-edge i8042
    3: 0 0 0 2 IO-APIC-edge
    4: 0 0 0 2 IO-APIC-edge
    7: 0 0 0 0 IO-APIC-edge parport0
    8: 0 0 0 0 IO-APIC-edge rtc0
    9: 0 0 0 0 IO-APIC-fasteoi acpi
    14: 4 9 509 975841 IO-APIC-edge pata_atiixp
    15: 0 0 0 0 IO-APIC-edge pata_atiixp
    16: 0 0 0 302 IO-APIC-fasteoi ohci_hcd:usb2, hda_intel
    17: 0 0 0 0 IO-APIC-fasteoi ohci_hcd:usb3, ohci_hcd:usb5
    18: 0 0 0 3 IO-APIC-fasteoi ohci_hcd:usb4, ohci_hcd:usb6, radeon
    19: 0 0 0 0 IO-APIC-fasteoi ehci_hcd:usb1
    22: 0 0 0 0 IO-APIC-fasteoi ahci
    23: 161880 1065333557 0 139 IO-APIC-fasteoi eth0
    NMI: 6003 7630 5800 5010 Non-maskable interrupts
    LOC: 409640264 557162611 369922896 333027728 Local timer interrupts
    SPU: 0 0 0 0 Spurious interrupts
    PMI: 6003 7630 5800 5010 Performance monitoring interrupts
    PND: 0 0 0 0 Performance pending work
    RES: 208984506 268127907 184089089 385358977 Rescheduling interrupts
    CAL: 4143 3603 4069 3271 Function call interrupts
    TLB: 11111763 11012884 10811417 10508745 TLB shootdowns
    TRM: 0 0 0 0 Thermal event interrupts
    THR: 0 0 0 0 Threshold APIC interrupts
    MCE: 0 0 0 0 Machine check exceptions
    MCP: 7779 7779 7779 7779 Machine check polls
    ERR: 0
    MIS: 0

  9.   wjens Says:

    I think I’m onto something here. Only one copy of emergent running at the time I did this, but this looks like an unusually large number of context switches, I think: [josiah@compute-1-11 ~]$ vmstat 1
    procs ———–memory———- —swap– —–io—- –system– —–cpu—–
    r b swpd free buff cache si so bi bo in cs us sy id wa st
    0 0 15536 4300340 184660 2826724 0 0 0 1 2 4 15 1 84 0 0
    0 0 15536 4300188 184660 2826728 0 0 0 0 29437 46260 0 8 92 0 0
    1 0 15536 4300244 184660 2826728 0 0 0 0 29331 46047 0 8 92 0 0
    1 0 15536 4300252 184660 2826728 0 0 0 0 29455 45942 0 9 91 0 0
    0 0 15536 4300308 184660 2826728 0 0 0 0 29331 45966 0 9 91 0 0
    1 0 15536 4300432 184660 2826728 0 0 0 4 29421 46022 0 9 91 0 0
    0 0 15536 4300480 184668 2826724 0 0 0 12 29476 45927 0 9 90 1 0
    2 0 15536 4300152 184668 2826728 0 0 0 0 29682 45958 0 11 89 0 0
    1 0 15536 4300296 184668 2826728 0 0 0 0 29472 46325 0 9 91 0 0
    1 0 15536 4300408 184668 2826728 0 0 0 0 28476 46944 0 11 89 0 0
    0 0 15536 4300064 184668 2826728 0 0 0 0 29595 46052 0 9 91 0 0
    1 0 15536 4300280 184668 2826728 0 0 0 0 29237 45805 0 10 90 0 0
    1 0 15536 4300160 184668 2826728 0 0 0 0 29699 46107 0 10 90 0 0
    0 0 15536 4300264 184668 2826728 0 0 0 0 29338 45889 0 9 91 0 0
    0 0 15536 4300440 184668 2826728 0 0 0 0 29482 46034 0 8 92 0 0
    2 0 15536 4300120 184668 2826728 0 0 0 0 29398 46069 0 7 93 0 0
    ^C
    [josiah@compute-1-11 ~]$
    As compared to a different machine with low CPU load:
    procs ———–memory———- —swap– —–io—- -system– —-cpu—-
    r b swpd free buff cache si so bi bo in cs us sy id wa
    0 0 0 1412380 1304676 3741928 0 0 2 5 5 4 0 0 99 0
    0 0 0 1412504 1304676 3741928 0 0 0 0 157 66 0 0 100 0
    0 0 0 1412496 1304676 3741928 0 0 0 0 110 67 0 0 100 0
    0 0 0 1412496 1304676 3741928 0 0 0 0 121 68 0 0 100 0
    0 0 0 1412496 1304676 3741928 0 0 0 0 189 209 0 0 100 0
    0 0 0 1412496 1304676 3741928 0 0 0 0 196 185 0 0 100 0
    0 0 0 1412496 1304676 3741928 0 0 0 0 146 73 0 0 100 0
    0 0 0 1412496 1304676 3741928 0 0 0 0 101 59 0 0 100 0

  10.   wjens Says:

    So the machine running one copy of emergent has tens of thousands of context switches per second, whereas bork (relatively idle) has between 0 and 209 in the time we looked at it.

  11.   wjens Says:

    For contrast, here’s a fully loaded machine with 16 copies of dnetc running:

    [josiah@compute-2-1 distributed-2-1]$ vmstat 1
    procs ———–memory———- —swap– —–io—- –system– —–cpu—–
    r b swpd free buff cache si so bi bo in cs us sy id wa st
    16 0 0 12574428 315436 10930952 0 0 0 0 3 1 2 0 97 0 0
    16 0 0 12574296 315436 10930952 0 0 0 0 14428 1131 88 0 13 0 0
    16 0 0 12574296 315436 10930952 0 0 0 0 14406 1133 88 0 13 0 0
    16 0 0 12574296 315436 10930952 0 0 0 0 14464 1147 88 0 12 0 0
    16 0 0 12574304 315436 10930952 0 0 0 0 14087 474 89 0 11 0 0
    16 0 0 12574304 315436 10930952 0 0 0 0 14219 767 87 0 13 0 0
    16 0 0 12574304 315436 10930952 0 0 0 0 14231 757 87 0 13 0 0
    16 0 0 12574304 315436 10930952 0 0 0 0 14416 1137 89 0 11 0 0
    16 0 0 12574304 315436 10930952 0 0 0 0 14431 1167 86 0 14 0 0
    16 0 0 12574304 315436 10930952 0 0 0 0 14362 1074 89 0 11 0 0
    16 0 0 12574304 315436 10930952 0 0 0 0 14372 1074 93 0 7 0 0
    16 0 0 12574304 315436 10930952 0 0 0 0 14360 1074 82 0 18 0 0
    16 0 0 12574304 315436 10930952 0 0 0 0 14367 1072 93 0 7 0 0
    16 0 0 12574428 315440 10930952 0 0 0 12 14367 1093 82 0 18 0 0
    16 0 0 12574428 315440 10930952 0 0 0 0 14406 1168 86 0 14 0 0
    16 0 0 12574428 315440 10930952 0 0 0 0 14405 1162 88 0 13 0 0
    16 0 0 12574428 315440 10930952 0 0 0 0 14401 1140 87 0 13 0 0

    Only a little over 1000 context switches… so around 45x as many with only two copies of emergent running. This seems like perhaps it’s the problem… or it could be a red herring. Probably worth asking the emergent list about, at least?

  12.   wjens Says:

    Sorry, that’s ONE copy of emergent, not two

  13.   wjens Says:

    Some more food for thought: I have discovered two facts, and I’m not sure exactly what they mean. 1. those stuck emergent processes appear to be blocking on a read(), probably from standard input (perhaps a pipe), and each emergent process has around 100 FIFO pipes open. Example:

    [root@compute-4-4 ~]# ps aux | grep emergent
    jdavila 22084 0.0 0.0 161184 5856 ? Ss May01 0:57 python /home/jdavila/run_parallel_experiments_here/jaimes_little_emergent_pusher.py
    jdavila 29485 7.8 1.0 4259272 358280 ? Sl 23:35 0:01 emergent -a . -ap . -nogui -ni -proj xorForBreveTake3.proj batches=2 epoch=1 log_dir=.
    jdavila 29486 7.8 1.0 4259428 358400 ? Sl 23:35 0:01 emergent -a . -ap . -nogui -ni -proj xorForBreveTake3.proj batches=2 epoch=1 log_dir=.
    jdavila 29699 9.5 1.0 4259300 358320 ? Sl 23:35 0:01 emergent -a . -ap . -nogui -ni -proj xorForBreveTake3.proj batches=2 epoch=1 log_dir=.
    root 29750 0.0 0.0 105292 868 pts/0 S+ 23:35 0:00 grep emergent
    [root@compute-4-4 ~]# strace -p 29699
    Process 29699 attached – interrupt to quit
    read(0,
    +++ killed by SIGKILL +++
    [root@compute-4-4 ~]# ps aux | grep emergent
    jdavila 22084 0.0 0.0 161184 5856 ? Ss May01 0:57 python /home/jdavila/run_parallel_experiments_here/jaimes_little_emergent_pusher.py
    jdavila 29766 9.5 1.0 4259408 358404 ? Sl 23:37 0:01 emergent -a . -ap . -nogui -ni -proj xorForBreveTake3.proj batches=2 epoch=1 log_dir=.
    jdavila 30301 24.4 1.0 4259268 360320 ? Sl 23:37 0:01 emergent -a . -ap . -nogui -ni -proj xorForBreveTake3.proj batches=2 epoch=1 log_dir=.
    jdavila 30448 74.5 1.0 697776 342328 ? R 23:37 0:01 emergent -a . -ap . -nogui -ni -proj xorForBreveTake3.proj batches=2 epoch=1 log_dir=.
    root 30450 0.0 0.0 105292 868 pts/0 S+ 23:37 0:00 grep emergent
    [root@compute-4-4 ~]# lsof | grep 30448 | wc -l
    200
    [root@compute-4-4 ~]# lsof | grep 30448 | grep pipe | wc -l
    101
    [root@compute-4-4 ~]#

  14.   wjens Says:

    Building a new version with ./configure –clean –debug –prefix=/share/apps –emer-mac-arch-bits=64

    Had to remove #define dDOUBLE from /share/apps/include/ode/common.h, which I had to put there to get breve to build.

  15.   wjens Says:

    Could this have anything to do with things being all wrong? Incorrect precision? The error when building emergent said that I couldn’t define dSINGLE and dDOUBLE both, just one, meaning emergent had defined dSINGLE… but I chose to define dDOUBLE when building breve (randomly), because it didn’t seem to define either, which seems wrong. I suppose I could rebuild breve with dSINGLE defined instead… but this seems unlikely, to me, to be the problem… though I don’t really know enough to have that opinion.

  16.   wjens Says:

    New emergent with debugging built and ready to install. Waiting for Jaime’s go-ahead.

  17.   jjdccs Says:

    Go for it. Thanks!

  18.   wjens Says:

    Installed.

  19.   wjens Says:

    So it was waiting on user input! Apparently it was asking if we wanted to save something on exit, which it shouldn’t have been doing in non-interactive mode. Still unclear why it didn’t do that on Jaime’s laptop, or why it didn’t do it if you only ran one copy of it, but one of the developers could tell what was up from the debugging output, commited a fix, I pulled it down, recompiled, installed, and problem solved! Yay! Case closed. Run times went from several hours down to ~30 minutes. Very nice.

Leave a Reply

You must be logged in to post a comment.