u/compgeek38400

Sorry about my abscense, I've been on my vacation/holiday.

I've done a lot of learning the last few weeks. Here are the highlights

1. I'm completely redoing my memory management. I'm working on a Physical Memory Manager (PMM), AND a virtual memory manager (VMM). Completely redoing how I deal with paging. I came to realize that except for reserved and my video physical memory, I really don't care where it gets assigned. Also, again except for video and reserved physical memory, it can all be swapable.

My PMM keeps track of the following items: a) does the memory exist; b) is it available, for example I define my video memory as not available, it is allocated early on, and I don't want the PMM deallocating it; c) Is it allocated to a virtual address yet; d) in concert with c, is it sharable (multi-allocated); e) is it swappable, I don't want to swap video and reserved memory for example.

My VMM keeps track of the following: a) is it swapped out (future); b) is it read only, such as my kernel code and .rodata; c) is it kernel memory, not sure if I really need this, but I had spare bits, I might want to use this in my swapping algorithm; d) is it shared.

Much of what I had in my paging .c file, has moved to my PMM and VMM. I'm hoping to finish this this week.

Once I have those three areas done, I will redo my heap and then make my github code public.

I'd also like some advice, I have two things I'm thinking about doing next: 1) adding threading, I'm not actually sure what I'd do with it right now as I don't really have a block device to save swapped blocks to; 2) adding support for user space, this would allow me to put things like most of my video and my shell into user space, this would also justify doing threading next; or 3) adding support for SATA, this would justify threading (for swap), but I might want to put it in user space then it would be restartable.

As always your opinions are always welcome.

Hopefully this will be useful for the people searching after me.

Thanks for reading!

I'm working on getting memory how I want it. I've moved my memory to the higher half. Now I'm ready for the final step. I want to move the page table entries I've made for the kernel to the area of physical memory I've designated for them.

I see several options.

1. since I know the physical addresses, I can temporarily double map them to virtual memory and then copy them.

2. I can scan the page directory and tables to find the virtual addresses for them. If i can't find them fall back to #1

3. I can re-make them where I want them.

4. other ideas?

I want to move them to a specific virtual address so I can exclude them from paging, having a designated eases the overhead of doing this. I can say if address > x do not page, or in reality only make virtual addresses between x and y pageable (my user space).

TIA for your input

After a week off for the family reunion, I'm back at writing my kernel. Here is a short summary of what I've learned:

1. DMA involves PHYSICAL memory addresses, it doesn't matter what the virtual address is. I probably should have figured this out on my own, because I remember having to change jumpers on the cards back in the 80's in order to change the DMA addresses. This calls into question the current memory map I'm working with, but for now I will leave it the same as I can't decide where I want to put my page tables. I know the industry standard is 'all over', or at least I've been told that. But I would prefer to have a dedicated area, it just seems like it would make everything easier keeping track of things. More to come on this subject.

2. I was working on my strtok() function and discovered __rawmemchr(). In reading the description, it says something like 'when the programmer knows that the character will exist'. After 40 years as a developer I know that these are just the kind of assumptions that eventually turn into bugs. I finally wrote my own strtok() because I didn't like gcc's use of __rawmemchr(). Probably personal preference.

3. Again as I was working on my strtok() I realized (yes I was copying) that they override the 'const' on the string parameter. I have all warnings set on when I compile, and treat warnings as errors. So they wouldn't even compile. After thinking long and hard about it, I decided that yes, overriding 'const' is a bad idea, so my strtok has the following definition

   char* strtokbuf(const char * const s, const char * const delims, char* buff, int buflen, int start, int *newstart)

This way I don't have to violate const.

My current issue is that when I set up my heap, I am getting a page fault. I tracked it back to I place my heap after my kernel code, but when I changed my memory map I don't actually allocated physical memory after the kernel. This lead to it is time to read my command line and find out how much space I want to allocate, hence strtok to parse my command line. I'm hoping to get my heap done by the end of the week.

Hope this helps some others while they are learning.

This is just an observation. I just think UIs are much less of the OS than say scheduling, a kernel, or disk subsystem.

Am I wrong?

This is the 7th part of what I'm learning writing a kernel, in pursuit of eventually writing an OS. I hope it will help others from making some of the same mistakes I did.

This week I decided I needed to redo how I do paging, I moved my page tables and set up the page directory so they would be right after the DMA paging area above the 16M mark. I also decided to do some major redesign. So I started over from scratch. Luckily much of I wrote after virtual memory and paging will be able to come over intact.

For those that are wondering this is the memory map I will be using:

JakelynnOS Memory Map
Virtual Address
Start	End
0x00000000	0x00010000	Identity Map
0x00000000	0x00001000	SEPT (System Entry Point Table)
0x000A8000	0x000BFFFF	Video Area (we use 0xB80000-0xBFFFF)
0x00010000	0x00FFFFFF	DMA Memory
0x01000000	0x0103FFFF	Page Table Entries (1024 of them)
0x01040000	0xBFFFFFFF	User Memory Area
0xC0000000	0xFFFFFFFF	Kernel Area

I think it will be much cleaner having one contiguous block from 0x01040000-0xBFFFFFFF to map into, but we shall see. Here is the memory map as returned by multiboot2:

multiboot2 memory map

I'm not sure yet what I'm going to do with the reserved memory in the 0x7xxxxxxx and 0xBxxxxxxx areas. Area 4 & 5 may be problematic depending on what I find there. I'm kind of hoping I can just ignore them, meaning mapping over the top of them.

I rewrote my process to map memory pages, I no longer had to worry about creating a page directory entry.

I collected much more multiboot info, not just memory and the memory map, but also the command line, boot loader name, BIOS boot device, ACPI RSDP info, and the Frame buffer info.

Once I get paging and the heap done, I will publish the github reference.

I also discovered how hard it is for me to keep the virtual/physical memory straight LOL

I will be on vacation/holiday this week at a family reunion so I doubt I'll be learning much new.

I hope you find this useful. Enjoy writing your own kernels and OSs.

When I changed my QEMU conf file to include disks and such I broke my memory map. This obviously is a design flaw that I will deal with this week.

But it raised a question. On the reserved memory returned by multiboot2 (I use GRUB2), should I identity map these areas?

Thanks in advance

A recent post got me thinking. Doing paging right requires a file system of some sort.

Do you think the paging file system belongs in user space, or kernel space. I am quick coming up on making such a decision myself

Continuing my series for those who are interested, if any. My source is here: https://github.com/tedavids/DragonOS

This was a long week, the first thing I learned put me back several days. Those of you following know this week I was finishing up my heap, allowing multi-page allocations.

This is what I learned:

1. Visual Studio Code when you 'Rename Symbol' it changes it EVERYWHERE in your project. This caused me to have to go back and redo code, and eventually just copy it back from Git, and start over.

2. I can't keep the whole project in my head anymore. I don't know if this is because of age, or because of the complexity of writing an OS. I hope it's the latter 😂

3. I'm going to have to take a couple of weeks to bring all my doco up to date.

I also have some questions:

Now that I have a working heap, I have lots of options on what to do next, these are some of the candidates:
a) multithreading -- I know this will entail at least creating atomic operations (atomic_t), sequencing stuff (mutex, semaphore), and retrofitting what I have to use them BEFORE I add this. What did I miss?

b) add a swap file system, I know this will involve writing a disk driver, should I do IDE/ATA or SCSI? If I did this, I'd be tempted to add both swap and a 'normal' file system.

c) something else?

I occasionally get page faults on startup. How would you debug this? I can't predict it, and never seem to get it when I am trying. How would you debug this? The only thing I can think of is to use line2addr, and hope

Thanks for reading, please chime in on what I should do next. I do know I'm going to spend this week updating doco. I may not change any code this week

In continuing letting those who care (if any) on what I'm learning making an OS, this is what I learned this week. My github link is here (https://github.com/tedavids/DragonOS).

This first thing I learned is the hardest part of writing a heap is deciding on the design. I read and reread the slab stuff found all over the net, and couldn't get my head around it. So I designed my own, when I look at this link (https://www.kernel.org/doc/gorman/html/understand/understand011.html) it looks similar to what I did but not quite. So some of it must have sunk in. This design is only appropriate for allocations under 1 page. I will do the page and above allocations this week. Probably using some sort of AVL binary tree structure, as I won't have to handle partial page allocations.

The second thing I learned is how much I've come to depend on malloc(), until you have a heap, you don't have malloc(). Free lists to the rescue.

Finally I learned how to make sub-bullets under bullet points in git. It makes the github readme easier for me to understand.

Thanks for reading.

I am working on physical memory management for my custom x86 32 bit kernel.

I have 5 functions: mapKernelPage, mapUserPage, unmapPage, setPageReadOnly, and setPageReadWrite.

Here is a code snippet from my kernel:

uint32_t physpage;
    uint32_t address = (uint32_t) mapKernelPage(0xC0158000, &physpage);


    printf("Successfully mapped page: 0x%Xl with physical address: 0x%Xl\n\r",
            address, physpage);


    printf("Testing Read only...");
    if (setPageReadOnly(address)) {
        printf("Success\n\r");
    } else {
        printf("Failed\n\r");
    }


    printf("Page 0x%Xl ", address);
    if (!unmapPage(address)) {
        printf("not unmapped\n\r");
    } else {
        printf("unmapped\n\r");
    }

I am testing via gdb. Before I do the map, my page is unavailable, afterwards it is. (I test my looking at memory), when I unmap the page it is again unavailable.

My issue is that when I set the page read only (it includes an invalidate page), I can still write to the memory.

Here is what I think is the pertinent info:

(gdb) info register cr0

cr0 0x80010011 [ PG WP ET PE ]

CR0 has the write protect bit set

Here is my page table entry:

(gdb) p/x pt[pte]

$7 = 0x1000001

Notice the read/write bit is zero

my page directory entry is:

(gdb) p/x page_directory[pde]

$13 = 0x114023

What am I missing?

TIA

From my reading the Page Table Entry sh ould override the Page Directory entry, what am I missing?

Thanks

I asked AI (I know, you all hate it but it has its place) how much faster a shift 12 bits (x >> 12) is than a divide by 0x1000. It came back 10 to 40 times. But also said modern compilers change a divide by 0x1000 to a shift.

I compiled and looked at the output, and darn if that isn't true.

So my question is, which do you do and why? To my mind either could be better depending on your audience. A divide is probably clearer to people that their first language isn't C (like me). But a shift would probably be clearer to someone that lives in C.

I look forward to both your flames and actual comments.

I've spent this week working on getting paging set up in my kernel. I have two bit arrays availMemMap, which tracks available physical memory, readwrtiteMap, which says what areas of memory are read/write (as opposed to read only, such as where the BIOS resides). I also now have the read/write bit in my page tables set appropriately. .text and .rodata are read only.

My main learnings are two things: a) when you try to write to a read only page, you don't get a GP fault, you get a page fault; and b) just because an area is 'reserved' in the multiboot info, does not necessarily mean it is read only, it might be, but not necessarily.

This coming week, I hope to get my page allocation/deallocation functions tested and working.

You can see my code (if you wish) here: https://github.com/tedavids/DragonOS

I do have a question. How to you allocate pages? Do you allocate kernel pages AFTER the kernel end, growing up. And user pages BEFORE the kernel, growing down? Or some other method? I'd be interested on hearing from those that have done this before.

Thanks for reading.

I have been working on a kernel for an OS, and after a month I have some lessons learned. Meaning what I would do differently if I was starting over from nothing. I'm posting this in the hope that it will hope someone else. I don't claim that this is the only, or even the best way. Just what I'd do if I was starting from scratch.

Firstly, I'd use OSDev.org as my guide. I'd join the forums and only after researching and failing for several days would I post a question. The regulars are really smart, and you don't want to wear out your welcome with questions you could have found in research.

Next I'd read the following sections: Introduction, Required knowledge, Beginner mistakes, Getting started, and how to ask questions. Especially Required knowledge, if you don't have it, get it before you start.

Next set up a Cross compiler (https://wiki.osdev.org/GCC_Cross-Compiler). Its really tempting to use the normal one, but DON'T

Once that is done do the bare bones tutorial. https://wiki.osdev.org/Bare_Bones Understand it then throw it away, and move on to the Meaty Skeleton Tutorial. https://wiki.osdev.org/Meaty_Skeleton

Once you understand it keep it for reference.

Next I'd move my kernel to the Higher Half. Tutorial here https://wiki.osdev.org/Multiboot_1_Higher_Half_x86_Bare_Bones

I'd start using what I learned in Meaty skeleton to start building my kernel. It is much easier to do it after you are in the Higher Half, than moving it. This is one place I started over.

Next I'd identity map (virtual address == real address) the lower meg of memory (0x0-0xFFFF). One thing I learned here is that the page directory table uses actual physical addresses, NOT virtual addresses.

In building the kernel this is the order I'd do things in after completing above:

1. Rewrite my snprintf routine to support hex, unsigned int, signed ints, boolean, hex, and string variables.

2. Make my GDT

3. Make my IDT, spend LOTS of time here, basically the GP (General Protection) fault, and Page Fault entries will become irreplaceable debugging tools. Really build them out. As part of the IDT also map the APIC ( https://wiki.osdev.org/APIC_Timer ), I kept getting random GP faults until I did this.

4. At this point I'd go back and move to Multiboot2, you will need a memory map for memory setting up useful paging and Multiboot2 can provide this to you. When you do this you will have to change QEMU from using "-kernel mykernel.abc" to "-cdrom mykernel.iso", be prepared to have this take some time. Once I had multiboot2, I'd extract the info to my own structure.

5. Next I'd make a keyboard handler https://wiki.osdev.org/PS/2_Keyboard#Commands This task isn't that hard but it is VERY putzy.

This brings us to some random thoughts.

a. when GRUB2 hands control over to your kernel, it places the multiboot info address in %ebx, save it before you use the register.

b. Mask the APIC so it sends less spurious interrupts here is the code:

# mask the APIC until we get it set up

xor %ax, %ax

mov	0xFF, %al

out	%al, $0x21		# Mask all master PIC IRQs

out %al, $0xA1		# Mask all slave PIC IRQs

c. when you compile remove the -O2 flag from your compiling, it optimizes out stuff as you try to debug. Get to know your debugger well before you start your kernel. I use gdb

d. Once you really are stuck (for several days), ask for help.

That's all for now. You can find my code here: https://github.com/tedavids/DragonOS

I hope this helps someone else.

PS my next steps are:

Set up my paging structures, I plan on using two bit arrays, one holding what memory is available for paging, and the other is it read only or read/write. Then start on functions to map/unmap pages. And then start working on a heap.

I'm putting this here in the hope that it helps someone else.

You can view the page descriptor table, and page entry as a 1024x1024 array. Where each element is a virtual page in memory. Array element [0,0] is virtual address 0, and element [1023,1023] is whatever that multiplies out as.

I teach my students to view array elements are like buckets. In these particular buckets are several things. First the 'present' bit, if this is zero, the buckets is empty. If it is not, the bucket also has a REAL address, which is the Real page the virtual address 'lives' at. Finally the buckets contains a bit that tells you if you can write to this area, or not.

I know for many, of you, this is obvious. But it took me several days on osdev.org using the paging pages and the bare bones paging page looking at code to come to this revelation.

I hope it helps someone