Antranig Vartanian<p><strong>Enabling DTrace in Python on FreeBSD (again)</strong></p><p><a href="https://antranigv.am/posts/2025/02/dtrace-python-freebsd-again/" rel="nofollow noopener noreferrer" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">antranigv.am/posts/2025/02/dtr</span><span class="invisible">ace-python-freebsd-again/</span></a></p><p>Last night I was running our usual Greybeard AMA on <a href="https://wiki.freebsd.org/Discord/DiscordServer" rel="nofollow noopener noreferrer" target="_blank">FreeBSD’s Discord server</a>, when someone asked “I’ve been using Linux for years, but I also like FreeBSD. what can I do for FreeBSD, and what can FreeBSD do for me, as a Python Full Stack Developer?”</p><p>I started talking about FreeBSD Jails, ZFS, Boot Environments and more, but I also wanted to focus on DTrace. The ability to dynamically trace on production still sounds like magic to me. I know it isn’t, I’ve read the code and the papers, but the fact that I can ask the operating system questions on the fly, without recompiling, is amazing.</p><p>So when I was demoing DTrace, I also wanted to show the DTrace integrations with Python. Little did I know that the Python package on FreeBSD was not compiled with the <code>--with-dtrace</code> option.</p><p>As a sane person, I setup a Jail (using <a href="https://jailer.dev/" rel="nofollow noopener noreferrer" target="_blank">Jailer</a>, of course), cloned the FreeBSD ports tree and added the <code>--with-dtrace</code> option back. It did not compile.</p><p>The first issue that we encountered was the following:</p><pre>--- Include/pydtrace_probes.h ---dtrace: option requires an argument -- s</pre><p>ah yes, looking over the Makefile we see the following</p><pre>Include/pydtrace_probes.h: $(srcdir)/Include/pydtrace.d $(MKDIR_P) Include $(DTRACE) $(DFLAGS) -o $@ -h -s $< : sed in-place edit with POSIX-only tools sed 's/PYTHON_/PyDTrace_/' $@ > $@.tmp mv $@.tmp $@</pre><p>(you can also view the source <a href="https://github.com/python/cpython/blob/v3.11.11/Makefile.pre.in#L1454C1-L1459C99" rel="nofollow noopener noreferrer" target="_blank">here</a>)</p><p>As far as I can tell, the <code>$<</code> thingie does not work with BSD Make, so how about if we try using <code>gmake</code> instead?</p><p>I did the following changes to the <code>Makefile</code> in the FreeBSD Ports tree. Basically adding <code>gmake</code> into <code>USES=</code>.</p><pre> USES= compiler:c11 cpe ncurses pathfix pkgconfig \- python:${PYTHON_DISTVERSION:R},env readline shebangfix ssl tar:xz+ python:${PYTHON_DISTVERSION:R},env readline shebangfix ssl tar:xz gmake</pre><p>Now let’s try compiling again.</p><p>We get the following errors now:</p><pre>ld: error: undefined symbol: __dtraceenabled_python___function__entry</pre><p>Ah yes, linking issues.</p><p>After an hour of digging we learned that the <code>DTRACE_OBJS=</code> variable is set to… nothing. But it needs to be set to <code>Python/pydtrace.o</code>. I was not able to fix this issue properly (in <code>configure</code>, <code>configure.ac</code>, or whatever madness that GNU Autotools use), so I just changed the line manually in the <code>Makefile</code>.</p><p>Now let’s try compiling again.</p><p>We get the following error:</p><pre>./Python/sysmodule.c:223:24: warning: passing 'const char *' to parameter of type 'char *' discards qualifiers [-Wincompatible-pointer-types-discards-qualifiers]</pre><p>Someone in the Discord chat recommended that I disable the LTO (Link-Time Optimization) option.</p><p>I did <code>make clean</code>, I patched that line again, and here we go, one more time.</p><p>It’s compiling, it’s compiling, it’s compiling… aaaaand we’re done! Let’s do <code>make install</code></p><pre>===> Installing for python311-3.11.11===> Checking if python311 is already installed===> Registering installation for python311-3.11.11[python.srv0.hackerspace.am] Installing python311-3.11.11...</pre><p>good! now let’s try DTrace with Python!</p><p>In one terminal, I ran Python, and in another one I got the following</p><pre># dtrace -l -n 'python*:::' ID PROVIDER MODULE FUNCTION NAME85302 python8737 libpython3.11.so.1.0 sys_audit audit85303 python8737 libpython3.11.so.1.0 sys_audit_tstate audit85304 python8737 libpython3.11.so.1.0 _PyEval_EvalFrameDefault function-entry85305 python8737 libpython3.11.so.1.0 dtrace_function_return function-return85306 python8737 libpython3.11.so.1.0 _PyEval_EvalFrameDefault function-return85307 python8737 libpython3.11.so.1.0 gc_collect_main gc-done85308 python8737 libpython3.11.so.1.0 gc_collect_main gc-start85309 python8737 libpython3.11.so.1.0 PyImport_ImportModuleLevelObject import-find-load-done85310 python8737 libpython3.11.so.1.0 PyImport_ImportModuleLevelObject import-find-load-start85311 python8737 libpython3.11.so.1.0 _PyEval_EvalFrameDefault line</pre><p>Woohoo! Now I’m happy!</p><p>Okay, so what did we learn?</p><ul><li>There’s a little bit of GNUMake-ism in Python’s Makefile. Either we have to use <code>gmake</code> in FreeBSD when building Python or we need to submit an alternative to the upstream</li><li>Die GNU Autotools… I mean… <em>the</em> GNU Autotools usage in Python has an issue, where the generated Makefile does not set the <code>DTRACE_OBJS</code> correctly. Can anyone help with this? As a Pascal guy, reading GNU configure files makes me wanna die.</li><li>LTO is problematic. Solutions?</li></ul><p>The real question is, how can we enable DTrace by default in FreeBSD packages for Python. DTrace is one of our market advantages and we should find a way to enable it everywhere we can.</p><p>This was a fun Greybeard AMA, where we touched multiple parts of the system. Looking forward for next week!</p><p><em>That’s all folks…</em></p><p>Reply via email.</p><p><a rel="nofollow noopener noreferrer" class="hashtag u-tag u-category" href="https://antranigv.am/tags/debugging/" target="_blank">#debugging</a> <a rel="nofollow noopener noreferrer" class="hashtag u-tag u-category" href="https://antranigv.am/tags/dtrace/" target="_blank">#DTrace</a> <a rel="nofollow noopener noreferrer" class="hashtag u-tag u-category" href="https://antranigv.am/tags/freebsd/" target="_blank">#FreeBSD</a> <a rel="nofollow noopener noreferrer" class="hashtag u-tag u-category" href="https://antranigv.am/tags/gnu/" target="_blank">#GNU</a> <a rel="nofollow noopener noreferrer" class="hashtag u-tag u-category" href="https://antranigv.am/tags/python/" target="_blank">#Python</a></p>