DragonFlyBSD

On Thu, 06 Jun 2013 16:32:19 +0200, Eric Christeson via Redmine
<bugtracker-admin@leaf.dragonflybsd.org> wrote:

I first noticed it after building a quickworld from master on Monday,
June 3. The previous known good was Thursday, May 30.

My feeling (although I have no clear idea about what happened exactly) is
that it is a quickworld issue across the libc version bump. libc in master
is libc.so.8.

I didn't try a quickworld myself but all boxes I upgraded with a full
build do not have this issue.

Sascha

This issue also occurs when trying to run the openjdk7 bootstrap binary on DragonFly 3.5

The bootstrap has been built on a previous DragonFly release and thus depends on libc.so.7

readelf -a bootstrap/bin/java reports:

Dynamic section at offset 0xc5c8 contains 22 entries:
[...]
0x0000000000000001 (NEEDED) Shared library: [libc.so.7]
[...]

libc.so.7 is the only libc name present in the ELF binary. Nothing unusual so far.

Yet, ldd reports bootstrap/bin/java is linked to both libc.so.7 and libc.so.8:

bootstrap/bin/java:
libz.so.3 => /usr/lib/libz.so.3 (0x800828000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x800a3c000)
libc.so.7 => /usr/lib/libc.so.7 (0x800c53000)
libc.so.8 => /usr/lib/libc.so.8 (0x800f8d000)

$ ldd -a bootstrap/bin/java

/red/@@-1:00007/dports/java/openjdk7/work/bootstrap/bin/java:
libz.so.3 => /usr/lib/libz.so.3 (0x800828000)
libpthread.so.0 => /usr/lib/libpthread.so.0 (0x800a3c000)
libc.so.7 => /usr/lib/libc.so.7 (0x800c53000)
/usr/lib/libz.so.3:
libc.so.8 => /usr/lib/libc.so.8 (0x800f8d000)
/usr/lib/libpthread.so.0:
libc.so.8 => /usr/lib/libc.so.8 (0x800f8d000)

This issue is also visible with some binaries only linked to libc.8.

The zsh 3.6:x86:64 package is one of these:

$ cd fo<TAB>
_tags:36: failed to load module `zsh/computil': /usr/lib/libc.so.8: No space available for static Thread Local Storage

zsh/computil.so is a shared library, not linked to multiple versions of shared libraries:

ldd ./computil.so
./computil.so:
libiconv.so.3 => /usr/local/lib/libiconv.so.3 (0x800d68000)
libdl.so.0 => /usr/lib/libdl.so.0 (0x801064000)
libncursesw.so.6 => /usr/lib/libncursesw.so.6 (0x801265000)
libm.so.4 => /usr/lib/libm.so.4 (0x8014db000)
libc.so.8 => /usr/lib/libc.so.8 (0x80081d000)

It appears to me that nmalloc is ultimately the culprit here.
It sets the thr_mags structure as TLS data. Right now NZONES equals 72 since ZALLOC_ZONE_LIMIT is 16K. That means on x86-64, the thr_mags structure size is 2*8*72 + 8 + 4 = 1164 bytes. Together will a couple of other TLS variables, the TLS section size of libc is 1172 bytes. (For i386 it's "only" 588 bytes)

In comparison, FreeBSD libc TLS section is 17 bytes.

The dmalloc implementation also sets a structure as TLS storage, but it seems to be more modest (depends on size of slab_zone + 42 bytes)

The immediate solution is to increase the value of RTLD_STATIC_TLS_EXTRA from 256 to likely 1280 on x86-64 and 640 on i386.

The mallocs should be reviewed to see if setting these structures to TLS was the right thing to do. In any case, I think we're stuck with increased *TLS_EXTRA space until we don't need to support libc's with such big TLS sections. (FreeBSD *TLS_EXTRA section is 128 bytes). This is per library I believe, so an executable with 20 linked libraries needs to allocate, for example, 1280x20 bytes = 25kb of TLS space during dynamic linking.

TLS space increased by this commit:
http://gitweb.dragonflybsd.org/dragonfly.git/commitdiff/4b2454d14575fe90aff242d1b282429ae99e2fea

Upgrading to latest world will probably fix these TLS storage errors of the 3.4 packages.