initial memory

[[e-mail]]

When some/all com/exe receive control from MSDOS,
all available memory has been allocated and the onus
is on the executable to free it if it wishes to be able to
do some mallocs in the future.

What is the rationale for this, and what is the situation?

I have startup code below that resizes that memory, as
per rules I no longer remember. I'd like to remove this
code if possible because it hardcodes the segment
shift (4), and it prevents the code/data/stack being
placed in non-contiguous areas of memory.

Where do I stand?

Thanks. Paul.



; determine how much memory is needed. The stack pointer points
; to the top. Work out what segment that is, then subtract the
; starting segment (the PSP), and you have your answer.

mov ax, sp
mov cl, 4
shr ax, cl ; get sp into pages
mov bx, ss
add ax, bx
add ax, 2 ; safety margin because we've done some pushes etc
mov bx, es
sub ax, bx ; subtract the psp segment

; free initially allocated memory

mov bx, ax
mov ah, 4ah
int 21h

[[e-mail]]

That code shouldn't removed for TSRs and programs which need to spawn other
program.

Ok, this is generic startup code for C programs, which may
do a system().

So for all executables, all of memory is allocated until
released?

Do you think that ideally MSDOS 2.0 should have included
a call to auto-release extra memory?

PDOS/86 has the ability to add extra calls like that. But I
need some way of testing whether I am running under a
PDOS-compliant (or some other existing standard). I was
thinking of getting the MSDOS version number and if it
was 0.1 or something like that, then it meant additional
calls were available to find more things like manufacturer
and capability (extra INT 21H calls).

BFN. Paul.

[Mateusz Viste]

Post by ***@gmail.com
So for all executables, all of memory is allocated until
released?

AFAIK this is true only for COM files, where CS=DS=ES=SS.
This mechanism allows for a very easy way DOS can load these programs.
Just allocate a single 64K block, put the COM image there, prefixed
with a PSP, preset CS=DS=ES=SS and point SP to the end of block.
EXE-style loading, in contrast, is *way* more complex.

Post by ***@gmail.com
Do you think that ideally MSDOS 2.0 should have included
a call to auto-release extra memory?

There is no reliable way to know how much memory your program needs.
Only you know that.

Mateusz

[JJ]

Post by Mateusz Viste

Post by ***@gmail.com
So for all executables, all of memory is allocated until
released?

AFAIK this is true only for COM files, where CS=DS=ES=SS.

EXE files can be made to allocate the maximum available memory. e.g.
compiling a tiny model program into an EXE.

[Mateusz Viste]

Post by JJ

Post by Mateusz Viste

Post by ***@gmail.com
So for all executables, all of memory is allocated until
released?

AFAIK this is true only for COM files, where CS=DS=ES=SS.

EXE files can be made to allocate the maximum available memory. e.g.
compiling a tiny model program into an EXE.

"can be made", yes, through fiddling with exe headers, but then that's
done on purpose, and not a by-design behavior, right?

Mateusz

[Mateusz Viste]

In the case of Borlands Tasm it seems to be by design. And with no
option to override *I* had to "fiddle" the EXE headers. :-\

Still, the Tasm tool does it on purpose. It has choice, it just decides
oterwise. When dealing with COM files, there is no choice.

And pardon me saying so, but how is "by design" *not* "on purpose" :-)

The design of the COM-style loading led to simplistic stack
sizing and allocation. I doubt COM loading was designed with the
specific goal of allocating a full 64K of memory with stack at the end -
it was just a consequence of an earlier design choice.

In the case of EXE files, there are headers to deal with this
situation. A tool that sets the headers in a specific way does so on
purpose.

Mateusz

[R.Wieser]

Mateusz,

Post by Mateusz Viste
Still, the Tasm tool does it on purpose.

Thats how its normally done. Waiting for it to do it by accident could
take a loooong time. :-)

Post by Mateusz Viste
It has choice, it just decides oterwise.

You sound as if you think it made the wrong one. Which decision do you
think should it have made and on what grounds ?

Post by Mateusz Viste
When dealing with COM files, there is no choice.

Huh? I thought I was responding to a remark pertaining the EXE style
programs.

But yes, I know that with the COM format there isn't much choice in this
regard. As such I already removed it from consideration.

Regards,
Rudy Wieser

[Mateusz Viste]

Post by R.Wieser
You sound as if you think it made the wrong one. Which decision do
you think should it have made and on what grounds ?

I don't think it made the wrong one. Or the good one. I do not discuss
the choice itself, surely it was backed up with careful consideration. I
only point out that it *is* a choice of the tool you use.

Now, just wondering aloud: is there an obvious reason why
Tasm does not pre-set MAXALLOC to be the same as MINALLOC? In the case
of high-level compilers, having a big MAXALLOC allows the runtime
to provide near pointers to the program via malloc(), but I assume that
Tasm, being an assembler, does not provide such conveniences.

Isn't it because the EXE header is, in fact, not generated by Tasm,
but by the linker (tlink)? Tlink might not know whether or not the
objects it links together come from Tasm or something else (that might
include libc calls). If that would be vaguely correct, then maybe
there's some tlink switch that could make it behave differently in this
context?

Mateusz

[R.Wieser]

Mateusz,

Post by Mateusz Viste
Now, just wondering aloud: is there an obvious reason why
Tasm does not pre-set MAXALLOC to be the same as MINALLOC?

...

Post by Mateusz Viste
Isn't it because the EXE header is, in fact, not generated
by Tasm, but by the linker (tlink)?

Really ? You're suggesting that Tasm is some kind of culprit and than
"solve" it by telling us that its actually TLink who does it ? And with it
"answering" a question which was never asked ?

Nope, not going to play that game.

Goodbye.

Regards,
Rudy Wieser

[Mateusz Viste]

Post by R.Wieser
Really ? You're suggesting that Tasm is some kind of culprit

I never suggested that, you did. Almost exactly 6 hours ago.

Post by R.Wieser
and than "solve" it by telling us that its actually TLink who does it

You seem to be misreading me. I am not solving anything, merely
suggesting that perhaps the problem is not where you are pointing at.

Mateusz

[R.Wieser]

Mateusz,

Post by Mateusz Viste
I never suggested that, you did. Almost exactly 6 hours ago.

Than you misunderstood. When I said "Borlands Tasm" I was referring to the
package. The Tasm as well as TLink programs, as you need both to be able to
create an executable.

Post by Mateusz Viste
You seem to be misreading me. I am not solving anything,

Thats for sure.

Post by Mateusz Viste
merely suggesting that perhaps the problem is not where
you are pointing at.

And which "the problem" might that be ? I do not remember having a problem.
Just something that I like to have seen different.

Also, I don't think I pointed anywhere, as I do not even have a solid idea
what exactly causes the particular result.

... Which I tried to make clear in that "almost exactly 6 hours ago" post.
Or did you really think that I only looked at one of the two involved
programs for such an option ?

Regards,
Rudy Wieser

[[e-mail]]

In the case
of high-level compilers, having a big MAXALLOC allows the runtime
to provide near pointers to the program via malloc(),

Can you elaborate on this? I'm not familiar with how
other people implement their runtime libraries, but to
use a near pointer (offset only), the maxalloc won't
be able to be more than 64k, less the size of the stack,
less the size of the bss, less the size of the data section.

Thanks. Paul.

[Mateusz Viste]

to use a near pointer (offset only), the maxalloc won't
be able to be more than 64k, less the size of the stack,
less the size of the bss, less the size of the data section.

SS might be somewhere else in the small model. Your remark is perfectly
valid for the tiny model of course. MAXALLOC is a way (for the
compiler/programmer) to tell DOS "please try putting this program
somewhere in memory where extra space is available because I might want
to use such near memory". From within a C program, the usual way (for
the programmer) to reclaim this memory is via malloc/calloc.

I can only imagine that's why the tlink thing maxes out MAXALLOC: it
assumes it might be used as a memory pool for runtime C near
malloc/calloc calls. This is, however, for those like Rudy using tasm,
since in a constrained environment (last 64K block used by DOS to load
the program) they won't be able to obtain any memory via int 21h,48h.

A possible workaround is to pre-allocate the necessary buffers through
a table (or some other filler) within assembly. The downside of this
solution is that it will obviously make the on-disk size of the program
that much larger. And you must know the amount of memory you need in
advance, too.

Mateusz

[JJ]

Post by Mateusz Viste

Post by JJ

Post by Mateusz Viste

Post by ***@gmail.com
So for all executables, all of memory is allocated until
released?

AFAIK this is true only for COM files, where CS=DS=ES=SS.

EXE files can be made to allocate the maximum available memory. e.g.
compiling a tiny model program into an EXE.

"can be made", yes, through fiddling with exe headers, but then that's
done on purpose, and not a by-design behavior, right?
Mateusz

It's the default result of MASM as well as TASM. No command line switch
required. The EXE has to be made like that to make the binaries of source
code for old system not break the system.

[JJ]

Post by ***@gmail.com
So for all executables, all of memory is allocated until
released?

For EXE files, it's header tells DOS how much memory should be allocated -
which can be specific amount, or maximum available.

Post by ***@gmail.com
Do you think that ideally MSDOS 2.0 should have included
a call to auto-release extra memory?

No, because that may break compatibility with old programs which are not
aware of MCBs. The program may end up corrupting the MCB following the
current one.

Post by ***@gmail.com
PDOS/86 has the ability to add extra calls like that. But I
need some way of testing whether I am running under a
PDOS-compliant (or some other existing standard). I was
thinking of getting the MSDOS version number and if it
was 0.1 or something like that, then it meant additional
calls were available to find more things like manufacturer
and capability (extra INT 21H calls).
BFN. Paul.

You can use the DOS OEM number returned by AH=30h. Though, that service is
DOS 2.0+ only. For DOS 1.0, I don't think there's a CP/M specification to
provide ID for CP/M variants.

[[e-mail]]

Post by JJ

Post by ***@gmail.com
PDOS/86 has the ability to add extra calls like that. But I
need some way of testing whether I am running under a
PDOS-compliant (or some other existing standard). I was
thinking of getting the MSDOS version number and if it
was 0.1 or something like that, then it meant additional
calls were available to find more things like manufacturer
and capability (extra INT 21H calls).

You can use the DOS OEM number returned by AH=30h. Though, that service is
DOS 2.0+ only. For DOS 1.0, I don't think there's a CP/M specification to
provide ID for CP/M variants.

DOS 2.0+ is fine. Thanks for that.

I see that Freedos already has a number. I might make
contact with them. I really need a number that does not
represent a particular OEM but instead a "standard". And
I just realized that if MSDOS was theoretically updated to
follow the standard, they wouldn't be able to switch to that
new number. Hmmm. Maybe they could. Maybe everyone
who wished to follow the standard could switch to that
number, and then if they wanted the "true manufacturer"
they have to do another API call.

Any thoughts?

There are a few extensions I am after:

1. A memory allocation routine that takes a 32-bit number
of bytes and returns a far pointer. I might also have a flag
to say whether I want the memory to be allocated in the
first 1 MiB. This is for a theoretical processor where the
segment shift value can be changed from 4 to anything
up to 16, or the equivalent is being done using a selector
and GDT+LDT maxed out and lined up.

2. I want to know if there is a PDOS-generic style parameters
on the stack.

3. I want a routine that takes a far pointer and a 32-bit integer
and adds them together to produce a normalized far pointer
(huge pointer) as per current segment shift/selector rules.

4. I want a routine to get the command line.

I'm not 100% certain about this because I think I might have
to instead just create a new 8086 OS and not attempt to be
MSDOS-compatible.

BFN. Paul.

[JJ]

Post by ***@gmail.com
I see that Freedos already has a number. I might make
contact with them. I really need a number that does not
represent a particular OEM but instead a "standard". And
I just realized that if MSDOS was theoretically updated to
follow the standard, they wouldn't be able to switch to that
new number. Hmmm. Maybe they could. Maybe everyone
who wished to follow the standard could switch to that
number, and then if they wanted the "true manufacturer"
they have to do another API call.

Nowadays, OS detection by detecting OS features is more reliable than simply
reading the OS ID.

[[e-mail]]

Post by JJ

Post by ***@gmail.com
I see that Freedos already has a number. I might make
contact with them. I really need a number that does not
represent a particular OEM but instead a "standard". And
I just realized that if MSDOS was theoretically updated to
follow the standard, they wouldn't be able to switch to that
new number. Hmmm. Maybe they could. Maybe everyone
who wished to follow the standard could switch to that
number, and then if they wanted the "true manufacturer"
they have to do another API call.

Nowadays, OS detection by detecting OS features is more reliable than simply
reading the OS ID.

It's actually OS features rather than OS ID that I want. How do
you suggest I do that? I need to know if the new style memory
allocation routine exists etc.

Thanks. Paul.

[JJ]

Post by ***@gmail.com
It's actually OS features rather than OS ID that I want. How do
you suggest I do that? I need to know if the new style memory
allocation routine exists etc.
Thanks. Paul.

You'll have to try using a feature and check whether the OS supports it or
not. Though, standard/common features are likely already supported by most
DOS variants. So, undocumented features are better targets. Or check
features which are non standard, or specific to a DOS variant. Preferrably,
those which are actually used by the included tools of those DOS variants to
check whether they're running in their own OS or not.

Other things that can be used is the fact that while all DOS variants likely
to already support all standard DOS features, their implementations are not.
It could be that some DOS services have different behaviour/algorithm, or
give different result. Or the OS has different/unusual data placements/order
for e.g. BUFFERS, FILES, and standard devices such as AUX, CON, PRN, etc.

[R.Wieser]

Muta,

Post by ***@gmail.com
I have startup code below that resizes that memory, as
per rules I no longer remember.

...

Post by ***@gmail.com
mov ax, sp
mov cl, 4
shr ax, cl ; get sp into pages

When a COM runs SP is set to the end of the SS segment. As a COM has all
the segments overlapping its upto how big your program is how much stack you
are actually reserving. But looking at my own programs I don't think that
reserving 60KB stack for a 4KB program is normally called for.

Hence I set SP myself (sometimes even to the programs top at 0100h) and
instead use a LASTBYTE label to figure out how much room I actually need
calculated the # of pages from it :

- - - - - - - - - - - - - - - - - - - -
lea sp,[StackTop] ;Set new Stack-top

mov bx,(offset LASTBYTE)+000Fh ;number of bytes in program + round-up
mov cl,04h ;convert to pages
shr bx,cl ;/
mov ah,4ah ;resize program space
int 21h ;/
jc @@Main_Error ;ERROR : Out of Memory !
- - - - - - - - - - - - - - - - - - - -

As you might see, I also made sure I could actually allocate it, as its
possible a COM gets loaded in a memory fragment thats not actually large
enough to hold it and the "extra" memory it needs.

Post by ***@gmail.com
So for all executables, all of memory is allocated until
released?

Yes for a .COM style program, and "it depends" for a .EXE style one - as
that latter one has got a header wich specifies lots of stuff.

Post by ***@gmail.com
tlink (which is what I originally used) puts x'ffff' as the number
of paragraphs to allocate, meaning allocate all memory.

Indeed it does (using Borlands Tasm v4.1 and TLink v7.1). I've never been
able to find a "minimize footprint" switch or setting, which has irked me to
no end. You see, one of the fields in an .EXEs header is
"min_extra_paragraphs" , containing - you guessed it - exactly how much it
needs. As such I wrote a small program that copies that field into the
"max_extra_paragraphs" one (which is, in our case, the culprit), recalculate
the SS:SP value from it and put it into their respective fields.

From than on the OS does, for the .EXE, all the work for me, allowing me to
drop the "shrink my memory" code from it. :-)

Regards,
Rudy Wieser

[[e-mail]]

Post by ***@gmail.com
tlink (which is what I originally used) puts x'ffff' as the number
of paragraphs to allocate, meaning allocate all memory.

Indeed it does (using Borlands Tasm v4.1 and TLink v7.1). I've never been
able to find a "minimize footprint" switch or setting, which has irked me to
no end. You see, one of the fields in an .EXEs header is
"min_extra_paragraphs" , containing - you guessed it - exactly how much it
needs. As such I wrote a small program that copies that field into the
"max_extra_paragraphs" one (which is, in our case, the culprit), recalculate
the SS:SP value from it and put it into their respective fields.

Could you explain these min/max/ss:sp rules please?

I see from here:

http://blog.marcinchwedczuk.pl/a-closer-look-at-portable-executable-msdos-stub

e_cp: 0x0003 // Pages in file

e_minalloc: 0x0000 // Minimum extra paragraphs needed
e_maxalloc: 0xffff // Maximum extra paragraphs needed

e_ss: 0x0000 // Initial (relative) SS value
e_sp: 0x00b8 // Initial SP value

What I would guess is that e_minalloc is the size of the
bss and stack segment. e_cp is the size of the code and
data segments (not including bss).

And thus I don't know why you can't just copy the minalloc
value to the maxalloc.

Or maybe minalloc is ONLY the bss size, in which case the
maxalloc extends the BSS, and the stack is placed after that.
That would be surprising though, as it means when you
release the memory you create a hole. But maybe the
allocated memory doesn't include the stack in the first place.

Looking at the PDOS/86 code it looks like the stack size is
included, and you can't create a hole otherwise your stack
will be overwritten by a spawned program.

BFN. Paul.

[R.Wieser]

Mta (paul),

Post by ***@gmail.com
Could you explain these min/max/ss:sp rules please?

Perhaps. Its a while ago ...

Post by ***@gmail.com
What I would guess is that e_minalloc is the size of the
bss and stack segment.

Yup.

Post by ***@gmail.com
And thus I don't know why you can't just copy the minalloc
value to the maxalloc.

Thats exactly what I did.

But when you do that you also need to recalculate SS/SP, otherwise it most
likely will point far beyond your program. Which is not a good idea.

Post by ***@gmail.com
Looking at the PDOS/86 code it looks like the stack size is
included,

Yep. Its comprised of all data areas that have no ititial value.

Post by ***@gmail.com
and you can't create a hole otherwise your stack
will be overwritten by a spawned program.

I think we need to define what a "hole" is here. But yes, its never a good
idea use memory areas you do not own for either BSS or stack.

Post by ***@gmail.com
Or maybe minalloc is ONLY the bss size

Definitily not.

But why don't you just try to make a program in which you enlarge the code,
BSS and stack areas (several combinations) and look at how the fields in the
EXE header change ? I found the EXE header info on the web and that is
what I did to verify if I understood it right.

Regards,
Rudy Wieser

[[e-mail]]

Post by R.Wieser

Post by ***@gmail.com
And thus I don't know why you can't just copy the minalloc
value to the maxalloc.

Thats exactly what I did.
But when you do that you also need to recalculate SS/SP, otherwise it most
likely will point far beyond your program. Which is not a good idea.

If the SS/SP are disturbed it implies that the difference between
minalloc and maxalloc is put between the bss and stack.

In addition, it means when MSDOS is loading the program,
it would need to manipulate the SS itself, depending how
much memory between minalloc and maxalloc it was able
to allocate.

Post by R.Wieser
But why don't you just try to make a program in which you enlarge the code,
BSS and stack areas (several combinations) and look at how the fields in the
EXE header change ? I found the EXE header info on the web and that is
what I did to verify if I understood it right.

That's a good idea.

Thanks. Paul.

[R.Wieser]

Muta (paul),

Post by ***@gmail.com
If the SS/SP are disturbed it implies that the difference between
minalloc and maxalloc is put between the bss and stack.

Nope. Minalloc refers to to the end of the combined the bss and following
stack segments. Maxalloc can go, as you read yourself, way beyond that.

But I owe you an apology : I remember updating SS/SP after setting MaxAlloc
to MinAlloc, but a peek into my program (still have it!) showed I made it
optional. In case I did not define a stack segment at all (but just kept
some BSS space free for it), causing SS:SP to be Zero in the EXE header.
And that ofcourse doesn't really work. So, I than can force it to be set to
the end of the BSS segment itself.

IOW, for a program in which a stack segment has been defined you only need,
as you mentioned, to copy MinAlloc into MaxAlloc.

I hope I did not create to much of a confusion.

Post by ***@gmail.com
That's a good idea.

:-) Its the way I've had to, and still do figure out lots of stuff, which
sometimes makes you run into the darnest things. And even when full
documentation was/is available I always had/have the need to verify what was
said.

Regards,
Rudy Wieser

[R.Wieser]

Muta (paul),

... having said that, if you want to use memory models other than "tiny"
you could try to see if you could find the start of the _DATA segment from

I forgot all about the .MAP file. Even easier, as it already contains the
starting offsets of each segment.

Regards,
Rudy Wieser