[upc-users] Max Local Segment Size is too small
Paul H. Hargrove
PHHargrove at lbl.gov
Fri Aug 19 15:24:00 PDT 2011
The value you see is 2GB-2MB-4KB, and is the maximum value by default.
You can set environment variable GASNET_MAX_SEGSIZE to override this
default. This variable takes integer values with optional 'M' or 'G'
suffixes. So GASNET_MAX_SEGSIZE=8G requests an 8GB segment. You can
find this and other useful environment variables documented in GASNet's
Relative to the size you request, you will still probably loose 2MB for
internal uses. Additionally, there are limitations on segment size
imposed by the O/S and network drivers that may prevent you from getting
all of your node's memory used in the GASNet segment(s). Note also that
if you are using all 48 cores per node, then 48*2G is already 50% more
than the 64G/node you report. So, you may have trouble using larger
Once you have confirmed that setting GASNET_MAX_SEGSIZE yields the
desired segment size, you should be able to reconfigure GASNet with, for
instance, "--with-segment-mmap-max=8GB" to increase the default value
(to avoid the need to set GASNET_MAX_SEGSIZE).
On 8/19/2011 2:03 PM, Debjyoti Majumder wrote:
> I am using GASNet for implementing Coarray Fortran runtime. I find the
> maximum memory that can be attached (gasnet_attach) to be inadequate. Is
> there a way to attach more memory using 'fast' or 'large' configuration
> without sacrificing performance?
> Function gasnet_getMaxLocalSegmentSize() returns 2145382400 (~2GB) for
> both FAST and LARGE configuration.
> I am using an Opteron system with 64GB memory on each node. Each node
> has 48 cores (8 NUMA nodes, 4 sockets).
> I tried using:
> 2 nodes, each with 1 process, using Infiniband conduit.
> 1 node ( 2 procs) using SMP conduit.
> I got the same max segment size for both cases.
> In UPC, I can use 5GB (by using shared-heap option in upcrun). I built
> UPC with GASNet 'fast'. How does UPC attach more memory?
> Debjyoti Majumder
> HPCTools group
> University of Houston
> upc-users mailing list
> upc-users at lbl.gov
Paul H. Hargrove PHHargrove at lbl.gov
Future Technologies Group
HPC Research Department Tel: +1-510-495-2352
Lawrence Berkeley National Laboratory Fax: +1-510-486-6900
More information about the upc-users