[computer-go] Allocating lots of memory in C++?

[Mike Young]

You probably have your answer by now about why your huge stack var failed. Here's some more detail about how Windows treats the stack. Other OS'es will likely be similar.

The initial stack allocation is specified in the link step. You can change this value via flags. The default in Win32 is 1 MB. The smallest valid allocation is 64KB, the virtual page granularity. It starts on a 64kb boundary, for what that's worth. The OS maintains a small guard page beyond the end of this allocation. I don't recall its size. Accessing this guard page triggers a segmentation fault, causing the fault handler to extend the stack. Stack vars are typically much smaller than your huge array. It exceeded the limit of that guard page, and so did not trigger stack expansion. Allocating the large array had the effect of setting the stack pointer to the end of the local allocations. Since the stack pointer was then set to invalid memory, subsequent stack operations (pushing function call args, for example) failed with the stack fault.

Coming from a garbage collected world, I can understand your trepidation about dynamic allocations. It need not be a problem. The typical idiom in C++ is to clean up allocated resources in the destructor. The very simplest class to do that might look something like the following. There are much more robust implementations freely available. Their use is typical in large development, often obligatory to ensure proper cleanup. The Boost library's smart_ptr<> and scoped_ptr<> are particularly good. smart_ptr is reference counted; scoped_ptr would be roughly equivalent to something like: 

template <typename T>
class BigArray {
    typedef BigArray<T> SelfT;
public:
    BigArray(size_t numElements)
    : buf(new T[numElements])
    {}

    ~BigArray() { delete[] buf; }

    //-- define copy ctor and assigment to copy the array.
    // or make them private to disallow copy and assignment.
    //---------
    BigArray(const SelfT &);
    BigArray & operator=(const SelfT &);

    operator T*() { return buf; }
    operator const T*() const { return buf; }
    T& operator[](size_t idx)
    {    //-- require idx < numElements. Or add a runtime check.
        return buf[idx];
    }
    const T& operator[](size_t idx) const;
    ...
protected:
    T * buf;
};

typedef BigArray<long long> LongLongArray;


  ----- Original Message ----- 
  From: Peter Drake 
  To: computer-go 
  Sent: Tuesday, August 15, 2006 12:48 PM
  Subject: Re: [computer-go] Allocating lots of memory in C++?


  Incidentally, this is stack allocation, not malloc or new.


  Peter Drake
  Assistant Professor of Computer Science
  Lewis & Clark College
  http://www.lclark.edu/~drake/








  On Aug 15, 2006, at 10:47 AM, Peter Drake wrote:


    I need to statically allocate an enormous block of memory (hundreds of megabytes) for Orego to store the search tree. When I allocate 1000 nodes, the program works fine. When I try to allocate 10000 (around 15MB), I get SIGSEGV. How do I crank up the allowable memory?


    Peter Drake
    Assistant Professor of Computer Science
    Lewis & Clark College
    http://www.lclark.edu/~drake/








    _______________________________________________
    computer-go mailing list
    computer-go at computer-go.org
    http://www.computer-go.org/mailman/listinfo/computer-go/




------------------------------------------------------------------------------


  _______________________________________________
  computer-go mailing list
  computer-go at computer-go.org
  http://www.computer-go.org/mailman/listinfo/computer-go/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://computer-go.org/pipermail/computer-go/attachments/20060816/377c5693/attachment.html