|
| dekhn wrote:
| I once complained about malloc happily allocating memory that the
| physical memory system couldn't satisfy (never let your hand
| write a check your ass can't cash?) but the more experienced
| programmer asked me if I'd heard of fractional reserve banking,
| and if not, whether it bothered me too.
| salawat wrote:
| If you're smart, the answer is yes, over reliance on
| statistical multiplexing scares the shit out of you, because
| it's all fun and games till the check is due.
| pjc50 wrote:
| Swap space is the Federal Reserve of memory allocation.
| [deleted]
| lmilcin wrote:
| Not the same thing.
|
| malloc() can tell everybody it has the memory but when push
| comes to shove the OS will have to admit overbooking.
| underdeserver wrote:
| TIL, unless you explicitly disable memory overcommit, it can
| and will overcommit.
|
| This is crazy to me.
| lmilcin wrote:
| It is not. It is actually very useful and advantageous for
| many reasons.
|
| The disadvantage is the pretty shitty failure mode.
|
| But in practice, I have never seen an operating system that
| was not a toy that was able to reliably, gracefully handle
| out of memory condition.
| dekhn wrote:
| So, uh, how do you feel about fractional reserve banking?
| Nearly all banks worldwide practice it. Statistically, it's
| not impossible that the entire world financial system could
| collapse due to uncorrelated bank runs.
| _3u10 wrote:
| It is impossible. Only a moron of magnificent magnitude
| would fail to print additional cash to cover the run.
|
| The problems caused by the feds failure to lend to First
| Bank of America during the Great Depression are well
| understood by the central banks.
|
| What would likely happen is the overnight rate would go
| up to 12%, and additional money would be printed to cover
| withdrawals for the month or two most people would be
| willing to forgo 12% interest in a potentially
| inflationary economy.
| Spivak wrote:
| That's exactly what fractional reserve banking is.
| lmilcin wrote:
| Funny thing, you are right. I was thinking about something
| else I guess.
| zeusk wrote:
| What do you think is a bank run?
| q-big wrote:
| > I once complained about malloc happily allocating memory that
| the physical memory system couldn't satisfy (never let your
| hand write a check your ass can't cash?) but the more
| experienced programmer asked me if I'd heard of fractional
| reserve banking, and if not, whether it bothered me too.
|
| What if you are worried about both? ;-)
| andi999 wrote:
| This has nothing to do with C. The problem/feature is with the
| systemcall that does the allocation, and any language has to use
| it.
| baktubi wrote:
| I pooped and scooped the poop said the Malloy to me be free.
| Along came the scoop and scoop it went. Free it did. In the
| bucket it tumbled for the others to enjoy.
| jimbob45 wrote:
| Is your issue that the system call doesn't return enough
| diagnostic information? If so, how would you have done it
| differently? I'm asking out of curiosity, not out of a
| reflexive instinct to defend C (which has many problems).
| joosters wrote:
| The difficulty is that the lack of memory might be discovered
| days after it was allocated (an over-committing allocator
| simply doesn't know at the time whether there will be memory
| or not) - how do you asynchronously tell the program that
| this allocation has now failed in a high-level way?
|
| Generally, UNIX will let you know about the missing memory by
| sending the process a signal. But by that point, there's not
| much that can be done to fix things up - and remember, all
| the fix up code would have to run without allocating any more
| memory itself. That's extremely tricky in C, and nigh-on
| impossible in other languages.
| not2b wrote:
| On Linux, people should be pointed to
|
| https://www.kernel.org/doc/Documentation/vm/overcommit-accou...
| DougN7 wrote:
| Is this really an issue with C? Isn't it ultimately an OS config
| issue?
| hdjjhhvvhga wrote:
| This not "In C", this is a system-related issue. I always disable
| overcommit as I see no benefit on my systems.
| Arnavion wrote:
| I used to disable overcommit, but I ran into a bunch of
| compilers that would crash because of it despite having 30GiB
| of free (as reported by free(1) ) memory available.
| lmilcin wrote:
| One of my interview questions starts with "Can a program allocate
| more memory than is physically available on the server?"
|
| Everybody gets this wrong (which is funny for a binary question)
| but it starts an interesting discussion through which I hope to
| learn how much they know about OS and virtual memory.
| floxy wrote:
| Don't leave us hanging. The "obvious" answer would seem to be
| "yes" because of swap. But if everyone gets that wrong...
| yjftsjthsd-h wrote:
| No, it's worse than that - the answer is "yes", because
| virtual memory + overcommit means that most of the time the
| OS will happily allow you to allocate more memory than
| physical+swap, and essentially gamble that you won't actually
| need all of it (and this is implemented because apparently
| that's almost always a winning bet).
| lmilcin wrote:
| Yeah. And the issue is that the actual problem happens
| sometime later when the application actually tries to use
| that memory. So you replaced an error that is relatively
| simple to handle with something that is impossible to
| handle reliably.
|
| So the operating system very much doesn't like to admit it
| doesn't have physical memory to back the area you are
| trying to use. Now it does not have a simple way to signal
| this to the application (there is no longer an option to
| return an error code) and so either everything slows down
| (as OS hopes that another process will return a little bit
| of memory to get things going for a little while) or one of
| the processes gets killed.
| OldHand2018 wrote:
| Processes come and go!
|
| The OS doesn't have to gamble that you won't actually need
| all the memory you allocate, it could just be a gamble that
| another memory hogging process exits before you need to use
| _all_ of your memory, or that you don 't need to use _all_
| of your memory _at the same time_.
| haxorrr666 wrote:
| Nope. Works fine on Fedora 34:
# free -h total used free
| shared buff/cache available Mem: 31Gi 3.1Gi 2.2Gi 27Mi 25Gi 27Gi
| Swap: 15Gi 62Mi 15Gi # uname -a Linux athena
| 5.13.19-200.fc34.x86_64 #1 SMP Sat Sep 18 16:32:24 UTC 2021
| x86_64 x86_64 x86_64 GNU/Linux # gcc -o memaloc memaloc.c #
| ./memaloc error!
| mirashii wrote:
| I'm a little disappointed that the article didn't answer the
| question, or at least try to. A discussion of using read/write vs
| mincore vs trying to catch a SIGSEGV would've been a nice
| addition.
| kibwen wrote:
| The answer is that you kind of can't. You're at the mercy of
| the OS to give you accurate information, and malloc as an
| interface isn't set up to distinguish between virtual memory
| and "actual" memory. We could imagine a separate interface that
| would allow the OS to communicate this distinction (or hacks
| like you allude to), but I don't know of any standard approach.
| convolvatron wrote:
| before linux - every unix OS would fail a page allocation if
| there wasn't a backing store. full stop.
|
| this worked really well
| davidw wrote:
| This is why some embedded systems don't do malloc (IIRC...been a
| while since I've read much about those).
| edwinbalani wrote:
| That's right - some "safe" coding standards like MISRA C go as
| far as forbidding use of malloc() without explicit
| justification.
|
| If you still need dynamic allocation, you might choose to have
| a custom allocator working from a fixed-size pool or arena
| created in code (possibly itself carved out of RAM with
| malloc(), but importantly only once at first setup, where you
| have a better guarantee that the allocation will succeed).
| edwinbalani wrote:
| (All that said, embedded systems sometimes don't have virtual
| memory, so the original problem stated in the link is just
| not a thing...)
|
| > working from a fixed-size pool or arena created in code
| (possibly itself carved out of RAM with malloc(), but
| importantly only once at first setup, where you have a better
| guarantee that the allocation will succeed)
|
| And I should add to this that you probably want to access all
| of the pool/arena to do setup, or just ensure it's physically
| allocated if you _are_ running in a virtual memory space.
| This is something that is reasonable at setup time, though.
| stefan_ wrote:
| Embedded systems (the ones where you would disallow malloc)
| don't generally have virtual memory by virtue of having no MMU,
| so on those you can't do overcommitment since there is no page
| fault mechanism.
|
| No, the reason is simply that by statically allocating all
| memory you can avoid entire classes of program faults and bugs.
| There are no memory leaks and you don't need to solve the NP-
| complete problem of "is there an execution path where a dynamic
| memory allocation will fail". Keep in mind that it is not just
| about the total amount of dynamically allocated memory, but
| also the order of allocations (and frees).
| tjoff wrote:
| I wouldn't say this is the reason. Embedded systems typically
| don't have virtual memory to start.
|
| I would expect (but verify) a malloc implementation on embedded
| to return null if it can't satisfy the allocation.
|
| But even with that assumption malloc in embedded is often a bad
| idea. You need to plan for worst case anyway and you can not
| afford memory fragmentation.
| haxorrr666 wrote:
| Nope. Works fine on Fedora 34:
|
| # free -h total used
| free shared buff/cache available
|
| Mem: 31Gi 3.1Gi 2.2Gi 27Mi 25Gi 27Gi
|
| Swap: 15Gi 62Mi 15Gi
|
| # uname -a
|
| Linux athena 5.13.19-200.fc34.x86_64 #1 SMP Sat Sep 18 16:32:24
| UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
|
| # gcc -o memaloc memaloc.c
|
| # ./memaloc
|
| error!
| yjftsjthsd-h wrote:
| TLDR: "You don't." (Because malloc hands you virtual memory and
| actually trying to use it might reveal that the system doesn't
| have the _real_ memory to handle your request.
|
| I kept reading hoping that there was going to be a solution, but
| not really; there are comments discussing disabling overcommit,
| but even that's a tradeoff (it _does_ fix this failure mode, but
| you might not want to actually run a system like that).
| one_off_comment wrote:
| There's gotta be some way to programmatically determine it,
| right? It may not be portable. It may require some system calls
| or something, but there's gotta be a way, right?
| wmf wrote:
| I guess you could mlock() your memory and see if that
| succeeds or fails. When overcommit is enabled no memory is
| really safe.
| yjftsjthsd-h wrote:
| I'm not an expert enough to tell you; I just read the article
| and decided that it was too long so summarized for others.
| There's some discussion upthread about catching SIGSEGV and
| other methods, FWIW.
| itamarst wrote:
| This difference between "malloc() succeeded and physical/swap
| memory is actually available" also has somewhat corresponding
| impact on how you measure memory usage.
|
| One approach is RSS, the memory in physical RAM... but what if
| you're swapping? Then again, maybe you swapped out memory you
| don't actually need and ignoring swap is fine.
|
| The other approach is "how much memory you allocated", and then
| you hit fun issues mentioned in this article, like "the OS
| doesn't actually _really_ allocate until you touch the page".
|
| (Longer version: https://pythonspeed.com/articles/measuring-
| memory-python/)
| _3u10 wrote:
| It's likely being OOM killed on Linux. Not sure what's happening
| on Mac. Try allocating a terabyte of swap and it should run.
|
| Alternatively use mmap & mlock to verify the allocation
| succeeded, but the process can still be OOM killed at any time
| for any reason.
| valleyer wrote:
| Similarly on macOS. There is a limit of 64 gigs in the VM
| compressor (in-core and on-disk compressed "segments"
| combined); when this is reached, a process that owns more than
| 50% of the compressed memory can be killed.
|
| See no_paging_space_action() in:
|
| https://opensource.apple.com/source/xnu/xnu-7195.81.3/bsd/ke...
|
| Edit: I think the 64 gigs number is out of date -- looks like
| it's now based in part on the amount of physical memory in the
| machine.
| _3u10 wrote:
| You can tell the FreeBSD / xnu devs take their job more
| seriously. A failure in the VM compressor sounds so much more
| professional than being OOM killed.
| pcwalton wrote:
| It's important to remember the biggest reason why overcommit
| exists on Linux and macOS: fork(). When a process forks, the vast
| majority of the child process's memory is safely shared with the
| parent process, due to copy-on-write. But a strict accounting
| would say that the total memory usage of the system has doubled,
| which is too conservative in most cases. Since forking is so
| common on Unix, overcommit ends up being necessary pretty
| quickly. Otherwise, fork/exec would stop working in any process
| using more than half of the system memory.
| jiveturkey wrote:
| Most processes that fork know that they are going to fork.
| Therefore they can pre-fork very early, so as to commit the
| memory when they only have a bare minimum allocated. Your
| typical daemon does this anyway.
|
| Some other type of process like an interpreter that can
| subshell out doesn't know how big the allocation is going to
| get, would have to pre-fork early on.
|
| In this way, you wouldn't "need" overcommit and the Linux
| horror of OOM. Well, perhaps you don't need it so badly.
| Programs that use sparse arrays without mmap() probably need
| overcommit or lots of swap.
| convolvatron wrote:
| wouldn't you just account the COW pages against the parent
| until they are copied?
|
| kicking the can down the road means there isn't any longer a
| reasonable correction (failing the allocation), but instead we
| get to drive around randomly trying to find something to kill.
|
| this is particularly annoying if you are running a service.
| there is no hope for it to recover - for example by flushing a
| cache. instead the OS looks around - sees this fat process just
| sitting there, and .. good news, we have plenty of memory now.
| pjc50 wrote:
| But what happens when the kernel needs to copy one and has
| run out of memory? You still get a random process killed.
|
| (I note that Windows has a different approach, with "reserve"
| vs "commit", but nobody regards that as a preferential reason
| for using Windows as a server OS)
| trhway wrote:
| >there is no hope for it to recover - for example by flushing
| a cache. instead the OS looks around - sees this fat process
| just sitting there, and .. good news, we have plenty of
| memory now.
|
| Overcommit was godsent in the times of expensive memory and
| when people used virtual memory on disk (so it will spill low
| use memory pages there instead of the kill). Of course these
| days with abundance of cheap memory and people not
| configuring virtual memory any more we get the situation you
| describe.
| secondcoming wrote:
| Indeed. It's totally dumb that the OS is allowed lie to you
| when you've asked for some resources. And because of this
| behaviour people no longer check for success from malloc(),
| etc, because of laziness. It's a bad situation.
| joosters wrote:
| Please don't blindly declare it 'totally dumb'. If you
| disallow overcommit, you can end up with a system that
| can't fork and run /bin/true, even if there are gigabytes
| of memory left.
|
| Both styles of memory allocation have their uses, and their
| drawbacks, but please understand them before declaring many
| OS designers as stupid and dumb.
| giomasce wrote:
| I suppose vfork might help with that (though I don't
| understand why they don't directly add fork_and_execve,
| it would seem much easier).
|
| Also, the problem with Linux is not having overcommit,
| but notv being able to choose when to overcommit and when
| not. Windows makes that easier, AFAIU.
| gpderetta wrote:
| They do. Man posix_spawn. It comes with its own dsl to be
| able to support a small subset of all operations that are
| often performed between fork and execv.
|
| Vfork is usually a better solution.
|
| Edit: and yes, I would love to be able to disable
| overcommit per process.
| OldHand2018 wrote:
| This paper [1] argues that fork needs to be deprecated in
| favor of posix_spawn() or other more modern solutions. It
| claims that, among many other reasons, that fork
| encourages overcommit and that programs such as Redis are
| extraordinarily constrained if overcommit is disabled -
| because of fork.
|
| [1] https://dl.acm.org/doi/10.1145/3317550.3321435
| joosters wrote:
| It's a weakness of the fork()+exec() model, for sure.
| However, creating a fork_and_execve() API is extremely
| tricky. Just think of all the innumerable setup options
| you would need to give it, e.g. what file handles should
| be closed or left open? What directory should it start
| in? What environment variables should be set - or
| cleared? And on and on and on...
|
| the flexibility of a separate fork() then exec() means
| you can set up the initial state of a new process exactly
| as you want, by doing whatever work is needed between the
| two calls. If you merge them into one, then you will
| never be able to encapsulate all of that.
| OldHand2018 wrote:
| Let's say you malloc some memory and the computer actually
| has everything available.
|
| Everything is great, up until some other process on your
| system does a fork bomb of an infinitely recursive program
| that allocates nothing on the heap. You've just got a whole
| lot of quickly growing stacks hoovering up your physical
| memory pages.
| alexgartrell wrote:
| Allocated-but-unavailable is a totally reasonable part of
| the memory hierarchy.
|
| Main Memory => zswap (compressed memory) => swap
|
| In this case, the pages may be logically allocated or not
| -- the assurance is that the data will be the value you
| expect it to be when it becomes resident.
|
| Should those pages be uninitialized, the "Swapped" state is
| really just "Remember that this thing was all zeros."
|
| We could do computing your way, but it'd be phenomenally
| more expensive. I know this because every thing we
| introduce to the hierarchy in practice makes computing
| phenomenally _less_ expensive.
| cturner wrote:
| "We could do computing your way, but it'd be phenomenally
| more expensive"
|
| It must be viable - Windows prevents overcommit. But it
| has slow child-process-creation (edit: previously said
| "forking"), and this steers development towards native
| threads which is its own set of problems.
|
| I had never previously joined the dots on the point
| pcwalton makes at the top of this thread. It is a
| dramatic trade-off.
| pjc50 wrote:
| Windows does not have fork(), apart from the hidden
| ZwCreateProcess;
| https://news.ycombinator.com/item?id=9653975
|
| For ages cygwin had to emulate it manually by hand-
| copying the process.
| pcwalton wrote:
| > wouldn't you just account the COW pages against the parent
| until they are copied?
|
| That's what Linux does. But how do you return ENOMEM when the
| copy does happen and now the system is out of memory? Memory
| writes don't return error codes. The best you could do is
| send a signal, which is exactly what the OOM killer does.
| wahern wrote:
| > Otherwise, fork/exec would stop working in any process using
| more than half of the system memory.
|
| Somehow Solaris manages just fine.
|
| And don't forget that swap memory exists. Ironically, using
| overcommit without swap is asking for trouble on Linux.
| Overcommit or no overcommit, the Linux VM and page buffer
| systems are designed with the expectation of swap.
| joosters wrote:
| Solaris does have this problem! If you have a huge program
| running, fork()img it can fail on Solaris, even if you only
| want to just exec a tiny program. The key way of avoiding
| this is to ensure you have lots and lots of swap space.
| pjc50 wrote:
| How _does_ Solaris handle this? Or Darwin?
| wahern wrote:
| Solaris has strict memory accounting; fork will fail if the
| system can't guarantee space for all non-shared anonymous
| memory. macOS has overcommit (all BSDs do to some extent,
| at least for fork), but it also automatically creates swap
| space so you rarely encounter issues one way or another in
| practice.
|
| fork and malloc can also fail in Linux even with overcommit
| enabled (rlimits, but also OOM killer racing with I/O page
| dirtying triggering best-effort timeout), so Linux buys you
| a little convenience at the cost of making it impossibly
| difficult to actually guarantee behavior when it matters
| most.
| _3u10 wrote:
| Even without overcommit swap makes the system work better as
| unused pages can be written to disk and that memory used for
| disk cache and I think can also help with defragmentation of
| memory not sure how that process actually works.
| dekhn wrote:
| The biggest reason overcommit exists is because it allows the
| system to operate more efficiently. The reality is most
| applications touch only some of the pages they allocate, and
| it's silly for the system to fail a malloc. Often times other
| expensive cleanup activities can be deferred (you don't really
| want to drop a handy directory entry cache just so an app can
| be sure it got physically backed memory for its request, 99.99%
| of the time).
|
| IIUC Linux was really the first OS to make overcommit so
| prominent. Most systems were a lot more conservative.
| joosters wrote:
| In theory, a system without overcommit can run just as
| efficiently, IF you have reserved huge amounts of swap space.
| As long as swap is available, the OS can do all the same COW
| and efficiency tricks. It's not the physically backed memory
| is the limiting factor, it's RAM+swap
| dekhn wrote:
| no, the kernel maintains other allocated objects (dentry,
| inode caches) that it can't swap out. Under memory
| pressure, those get dropped before application pages. See
| https://unix.stackexchange.com/questions/17936/setting-
| proc-... and
| https://unix.stackexchange.com/questions/111893/how-long-
| do-...
|
| I've found the linux memory system to be far too
| complicated to understand for quite some time, compared to
| what's documented in, for example, The Design and
| Implementation of The FreeBSD Operating System, for a more
| comprehensible system.
| joosters wrote:
| Those objects must exist with and without overcommit, I
| don't understand why they must make one less efficient
| than the other.
|
| (I'm talking in general here - not Linux specifically)
| dekhn wrote:
| dentry caches exist with and without overcommit, but you
| get higher cache hit rates with overcommit, because you
| flush them less recently. Depending on workload, this can
| matter a lot. It mattered more in the time of hard
| drives.
| joosters wrote:
| I'm sorry, but I'm still not following... why must a
| cache be (in theory) flushed more often without
| overcommit?
| dekhn wrote:
| if overcommit is disabled, then the system drops its
| internal caches (dentry cache, pagecache) to satisfy a
| memory allocation. Since most applications don't touch
| pages they allocate, that means the OS could have avoided
| dropping the caches. Since it did drop the caches, other
| parts of the system will then have to do more work to
| reconstruct the caches (looking up an dentry explicitly,
| or loading a page from disk instead of RAM).
|
| Everything I'm describing is about a busy server with
| heterogenous workloads of specific types.
| secondcoming wrote:
| Even the linux docs for it suggest to turn it on if you're
| working with sparse arrays, there's no mention of fork()
|
| https://www.kernel.org/doc/Documentation/vm/overcommit-accou...
___________________________________________________________________
(page generated 2021-11-05 23:00 UTC) |