wrapped up the unification of the virtual whitespace parsing system, switched to allowing the virtual whitespace to be toggled

4coder_API.html

@@ -116,7 +116,10 @@ Flags can be combined with bit or to specify a state with multiple modifiers.<br
     A 1 indicates dos endings "\r\n" and a 0 indicates nix endings "\n".<br><br></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>BufferSetting_Unimportant</span></span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The BufferSetting_Unimportant setting marks a buffer so that it's dirty state will be completely
     ignored.  This means the "dirty" star is hidden and the buffer can be closed without presenting an
     "are you sure" dialogue screen.<br><br></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>BufferSetting_ReadOnly</span></span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The BufferSetting_ReadOnly setting marks a buffer so that it can only be returned from buffer
-    access calls that include an AccessProtected flag.<br><br></div></div></div></div><hr><div id='View_Setting_ID_doc' style='margin-bottom: 1cm;'><h4>&sect;3.4.13: View_Setting_ID</h4><div style='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>enum View_Setting_ID;</div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'>A View_Setting_ID names an adjustable setting in a view.<br><br></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Values</i></b></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>ViewSetting_Null</span></span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>ViewSetting_Null is not a valid setting, it is reserved to detect errors.<br><br></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>ViewSetting_ShowWhitespace</span></span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The ViewSetting_ShowWhitespace setting determines whether the view highlights
+    access calls that include an AccessProtected flag.  By convention this means that edit commands that
+    should not be applied to read only buffers will not edit this buffer.<br><br></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>BufferSetting_VirtualWhitespace</span></span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The BufferSetting_VirtualWhitespace settings enables virtual whitespace on a buffer.
+    Text buffers with virtual whitespace will set the indentation of every line to zero.
+    Buffers with lexing enabled will use virtual white space to present the code with appealing indentation.<br><br></div></div></div></div><hr><div id='View_Setting_ID_doc' style='margin-bottom: 1cm;'><h4>&sect;3.4.13: View_Setting_ID</h4><div style='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>enum View_Setting_ID;</div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'>A View_Setting_ID names an adjustable setting in a view.<br><br></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Values</i></b></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>ViewSetting_Null</span></span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>ViewSetting_Null is not a valid setting, it is reserved to detect errors.<br><br></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>ViewSetting_ShowWhitespace</span></span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The ViewSetting_ShowWhitespace setting determines whether the view highlights
     whitespace in a file.  Whenever the view switches to a new buffer this setting is turned off.<br><br></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>ViewSetting_ShowScrollbar</span></span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>The ViewSetting_ShowScrollbar setting determines whether a scroll bar is
     attached to a view in it's scrollable section.<br><br></div></div></div></div><hr><div id='Buffer_Create_Flag_doc' style='margin-bottom: 1cm;'><h4>&sect;3.4.14: Buffer_Create_Flag</h4><div style='font-family: "Courier New", Courier, monospace; text-align: left; margin-top: 3mm; margin-bottom: 3mm; font-size: .95em; background: #DFDFDF; padding: 0.25em;'>enum Buffer_Create_Flag;</div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Description</i></b></div><div style='margin-left: 5mm; margin-right: 5mm;'>A Buffer_Create_Flag field specifies how a buffer should be created.<br><br></div><div style='margin-top: 3mm; margin-bottom: 3mm; color: #309030;'><b><i>Values</i></b></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>BufferCreate_Background</span> = 0x1</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>BufferCreate_Background is not currently implemented.<br><br></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>BufferCreate_AlwaysNew</span> = 0x2</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>When BufferCreate_AlwaysNew is set it indicates the buffer should be
     cleared to empty even if it's associated file already has content.<br><br></div></div></div><div><div><span style='font-family: "Courier New", Courier, monospace; text-align: left;'><span style='font-weight: 600;'>BufferCreate_NeverNew</span> = 0x4</span></div><div style='margin-bottom: 6mm;'><div style='margin-left: 5mm; margin-right: 5mm;'>When BufferCreate_NeverNew is set it indicates that the buffer should

4coder_default_bindings.cpp

@@ -394,6 +394,7 @@ default_keys(Bind_Helper *context){
     bind(context, 'u', MDFR_CTRL, to_uppercase);
     bind(context, 'U', MDFR_CTRL, rewrite_as_single_caps);
     bind(context, 'v', MDFR_CTRL, paste_and_indent);
+    bind(context, 'v', MDFR_ALT, toggle_virtual_whitespace);
     bind(context, 'V', MDFR_CTRL, paste_next_and_indent);
     bind(context, 'x', MDFR_CTRL, cut);
     bind(context, 'y', MDFR_CTRL, cmdid_redo);

4coder_default_include.cpp

@@ -24,12 +24,8 @@
 //
 
 static int32_t
-open_file(Application_Links *app,
+open_file(Application_Links *app, Buffer_Summary *buffer_out, char *filename,
-          Buffer_Summary *buffer_out,
+          int32_t filename_len, int32_t background, int32_t never_new){
-          char *filename,
-          int32_t filename_len,
-          int32_t background,
-          int32_t never_new){
     int32_t result = false;
     Buffer_Summary buffer =
         get_buffer_by_name(app, filename, filename_len,
@@ -58,11 +54,8 @@ open_file(Application_Links *app,
 }
 
 static int32_t
-view_open_file(Application_Links *app,
+view_open_file(Application_Links *app, View_Summary *view, char *filename,
-               View_Summary *view,
+               int32_t filename_len, int32_t never_new){
-               char *filename,
-               int32_t filename_len,
-               int32_t never_new){
     int32_t result = 0;
     
     if (view){
@@ -77,11 +70,8 @@ view_open_file(Application_Links *app,
 }
 
 static int32_t
-read_line(Application_Links *app,
+read_line(Application_Links *app, Partition *part, Buffer_Summary *buffer,
-          Partition *part,
+          int32_t line, String *str){
-          Buffer_Summary *buffer,
-          int32_t line,
-          String *str){
     
     Partial_Cursor begin = {0};
     Partial_Cursor end = {0};
@@ -128,6 +118,7 @@ init_memory(Application_Links *app){
     general_memory_open(&global_general, general_mem, general_size);
 }
 
+
 //
 // Helpers
 //
@@ -142,6 +133,7 @@ refresh_view(Application_Links *app, View_Summary *view){
     *view = get_view(app, view->view_id, AccessAll);
 }
 
+
 //
 // Buffer Streaming
 //
@@ -2662,6 +2654,14 @@ CUSTOM_COMMAND_SIG(decrease_line_wrap){
     buffer_set_setting(app, &buffer, BufferSetting_WrapPosition, wrap - 10);
 }
 
+CUSTOM_COMMAND_SIG(toggle_virtual_whitespace){
+    View_Summary view = get_active_view(app, AccessProtected);
+    Buffer_Summary buffer = get_buffer(app, view.buffer_id, AccessProtected);
+    
+    int32_t vwhite = buffer_get_setting(app, &buffer, BufferSetting_VirtualWhitespace);
+    buffer_set_setting(app, &buffer, BufferSetting_VirtualWhitespace, !vwhite);
+}
+
 CUSTOM_COMMAND_SIG(toggle_show_whitespace){
     View_Summary view = get_active_view(app, AccessProtected);
     view_set_setting(app, &view, ViewSetting_ShowWhitespace, !view.show_whitespace);
@@ -2670,13 +2670,13 @@ CUSTOM_COMMAND_SIG(toggle_show_whitespace){
 CUSTOM_COMMAND_SIG(eol_dosify){
     View_Summary view = get_active_view(app, AccessOpen);
     Buffer_Summary buffer = get_buffer(app, view.buffer_id, AccessOpen);
-    buffer_set_setting(app, &buffer, BufferSetting_Eol, true);
+    buffer_set_setting(app, &buffer, BufferSetting_Eol, 1);
 }
 
 CUSTOM_COMMAND_SIG(eol_nixify){
     View_Summary view = get_active_view(app, AccessOpen);
     Buffer_Summary buffer = get_buffer(app, view.buffer_id, AccessOpen);
-    buffer_set_setting(app, &buffer, BufferSetting_Eol, false);
+    buffer_set_setting(app, &buffer, BufferSetting_Eol, 0);
 }
 
 CUSTOM_COMMAND_SIG(exit_4coder){
@@ -3283,15 +3283,15 @@ static void
 set_fancy_compilation_buffer_font(Application_Links *app){
     Buffer_Summary comp_buffer = get_buffer_by_name(app, literal("*compilation*"), AccessAll);
     buffer_set_font(app, &comp_buffer, literal("Inconsolata"));
-}                                                       
+}
-                                                        
+
-CUSTOM_COMMAND_SIG(build_in_build_panel){               
+CUSTOM_COMMAND_SIG(build_in_build_panel){
-    uint32_t access = AccessAll;                        
+    uint32_t access = AccessAll;
     View_Summary view = get_active_view(app, access);
     Buffer_Summary buffer = get_buffer(app, view.buffer_id, access);
-                                                        
+    
     View_Summary build_view = get_or_open_build_panel(app);
-                                                        
+    
     execute_standard_build(app, &build_view, &buffer);
     set_fancy_compilation_buffer_font(app);
     

4coder_types.h

@@ -159,8 +159,14 @@ ENUM(int32_t, Buffer_Setting_ID){
     BufferSetting_Unimportant,
     
     /* DOC(The BufferSetting_ReadOnly setting marks a buffer so that it can only be returned from buffer
-    access calls that include an AccessProtected flag.) */
+    access calls that include an AccessProtected flag.  By convention this means that edit commands that
+    should not be applied to read only buffers will not edit this buffer.) */
     BufferSetting_ReadOnly,
+    
+    /* DOC(The BufferSetting_VirtualWhitespace settings enables virtual whitespace on a buffer.
+    Text buffers with virtual whitespace will set the indentation of every line to zero.
+    Buffers with lexing enabled will use virtual white space to present the code with appealing indentation.) */
+    BufferSetting_VirtualWhitespace,
 };
 
 /* DOC(A View_Setting_ID names an adjustable setting in a view.) */

4ed.cpp

@@ -41,12 +41,6 @@ typedef struct Command_Data{
     System_Functions *system;
     Live_Views *live_set;
     
-#if 0
-    // TODO(allen): eliminate this shit yo!
-    Panel *panel;
-    View *view;
-#endif
-    
     i32 screen_width, screen_height;
     Key_Event_Data key;
 } Command_Data;

4ed_api_implementation.cpp

@@ -710,13 +710,14 @@ Buffer_Get_Setting(Application_Links *app, Buffer_Summary *buffer, Buffer_Settin
     
     if (file){
         switch (setting){
-            case BufferSetting_Lex:          result = file->settings.tokens_exist;     break;
+            case BufferSetting_Lex:               result = file->settings.tokens_exist;     break;
-            case BufferSetting_WrapLine:     result = !file->settings.unwrapped_lines; break;
+            case BufferSetting_WrapLine:          result = !file->settings.unwrapped_lines; break;
-            case BufferSetting_WrapPosition: result = file->settings.display_width;    break;
+            case BufferSetting_WrapPosition:      result = file->settings.display_width;    break;
-            case BufferSetting_MapID:        result = file->settings.base_map_id;      break;
+            case BufferSetting_MapID:             result = file->settings.base_map_id;      break;
-            case BufferSetting_Eol:          result = file->settings.dos_write_mode;   break;
+            case BufferSetting_Eol:               result = file->settings.dos_write_mode;   break;
-            case BufferSetting_Unimportant:  result = file->settings.unimportant;      break;
+            case BufferSetting_Unimportant:       result = file->settings.unimportant;      break;
-            case BufferSetting_ReadOnly:     result = file->settings.read_only;        break;
+            case BufferSetting_ReadOnly:          result = file->settings.read_only;        break;
+            case BufferSetting_VirtualWhitespace: result = file->settings.virtual_white;    break;
         }
     }
     
@@ -825,6 +826,32 @@ DOC_SEE(Buffer_Setting_ID)
                     file->settings.read_only = 0;
                 }
             }break;
+            
+            case BufferSetting_VirtualWhitespace:
+            {
+                b32 full_remeasure = 0;
+                if (value){
+                    if (!file->settings.virtual_white){
+                        file->settings.virtual_white = 1;
+                        full_remeasure = 1;
+                    }
+                }
+                else{
+                    if (file->settings.virtual_white){
+                        file->settings.virtual_white = 0;
+                        full_remeasure = 1;
+                    }
+                }
+                
+                if (full_remeasure){
+                    i16 font_id = file->settings.font_id;
+                    Render_Font *font = get_font_info(models->font_set, font_id)->font;
+                    
+                    file_measure_character_starts(models, file);
+                    file_measure_wraps_and_fix_cursor(models, file, (f32)font->height,
+                                                      font->advance_data);
+                }
+            }break;
         }
         fill_buffer_summary(buffer, file, cmd);
     }

4ed_file.cpp

@@ -91,7 +91,8 @@ struct Text_Effect{
 struct Editing_File_Settings{
     i32 base_map_id;
     i32 display_width;
-    i32 dos_write_mode;
+    b32 dos_write_mode;
+    b32 virtual_white;
     i16 font_id;
     b8 unwrapped_lines;
     b8 tokens_exist;

4ed_file_view.cpp

@@ -9,8 +9,6 @@
 
 // TOP
 
-#define VWHITE 1
-
 internal i32
 get_or_add_map_index(Models *models, i32 mapid){
     i32 result = 0;
@@ -410,7 +408,7 @@ view_compute_cursor(View *view, Buffer_Seek seek, b32 return_hint){
     params.adv              = font->advance_data;
     params.wrap_line_index  = file->state.wrap_line_index;
     params.character_starts = file->state.character_starts;
-    params.virtual_white    = VWHITE;
+    params.virtual_white    = file->settings.virtual_white;
     params.return_hint      = return_hint;
     params.cursor_out       = &result;
     
@@ -972,7 +970,7 @@ file_allocate_character_starts_as_needed(General_Memory *general, Editing_File *
 internal void
 file_measure_character_starts(Models *models, Editing_File *file){
     file_allocate_character_starts_as_needed(&models->mem.general, file);
-    buffer_measure_character_starts(&file->state.buffer, file->state.character_starts, 0, VWHITE);
+    buffer_measure_character_starts(&file->state.buffer, file->state.character_starts, 0, file->settings.virtual_white);
     file_update_cursor_positions(models, file);
 }
 
@@ -1000,18 +998,196 @@ file_allocate_wrap_positions_as_needed(General_Memory *general, Editing_File *fi
                                      min_amount, sizeof(f32));
 }
 
+struct Code_Wrap_State{
+    Cpp_Token_Array token_array;
+    Cpp_Token *token_ptr;
+    Cpp_Token *end_token;
+    
+    f32 paren_nesting[32];
+    i32 paren_safe_top;
+    i32 paren_top;
+    
+    i32 *line_starts;
+    i32 line_index;
+    i32 next_line_start;
+    
+    f32 x;
+    b32 consume_newline;
+    
+    Buffer_Stream_Type stream;
+    i32 i;
+    
+    f32 *adv;
+    f32 tab_indent_amount;
+};
+
+internal void
+wrap_state_init(Code_Wrap_State *state, Editing_File *file, f32 *adv){
+    state->token_array = file->state.token_array;
+    state->token_ptr = state->token_array.tokens;
+    state->end_token = state->token_ptr + state->token_array.count;
+    
+    state->line_starts = file->state.buffer.line_starts;
+    state->next_line_start = state->line_starts[1];
+    
+    Buffer_Type *buffer = &file->state.buffer;
+    i32 size = buffer_size(buffer);
+    buffer_stringify_loop(&state->stream, buffer, 0, size);
+    
+    state->adv = adv;
+    state->tab_indent_amount = adv['\t'];
+}
+
+struct Code_Wrap_Step{
+    i32 position_start;
+    i32 position_end;
+    
+    f32 final_x;
+};
+
+internal void
+wrap_state_set_x(Code_Wrap_State *state, f32 line_shift){
+    state->x = line_shift;
+}
+
+internal Code_Wrap_Step
+wrap_state_consume_token(Code_Wrap_State *state, i32 fixed_end_point){
+    Code_Wrap_Step result = {0};
+    i32 i = state->i;
+    
+    result.position_start = i;
+    
+    if (state->consume_newline){
+        ++i;
+        state->x = 0;
+        state->consume_newline = 0;
+    }
+    
+    b32 skipping_whitespace = 0;
+    if (i >= state->next_line_start){
+        state->x = state->paren_nesting[state->paren_safe_top];
+        skipping_whitespace = 1;
+    }
+    
+    // TODO(allen): exponential search this shit!
+    while (i >= state->next_line_start){
+        ++state->line_index;
+        state->next_line_start = state->line_starts[state->line_index + 1];
+    }
+    
+    i32 line_start = state->line_starts[state->line_index];
+    
+    b32 still_looping = 0;
+    i32 end = state->token_ptr->start + state->token_ptr->size;
+    
+    if (fixed_end_point >= 0 && end > fixed_end_point){
+        end = fixed_end_point;
+    }
+    
+    if (i < line_start){
+        i = line_start;
+    }
+    
+    if (i == line_start){
+        skipping_whitespace = 1;
+    }
+    
+    do{
+        for (; i < state->stream.end; ++i){
+            if (!(i < end)){
+                goto doublebreak;
+            }
+            
+            u8 ch = (u8)state->stream.data[i];
+            
+            if (ch != ' ' && ch != '\t'){
+                skipping_whitespace = 0;
+            }
+            
+            if (!skipping_whitespace){
+                if (ch == '\n'){
+                    state->consume_newline = 1;
+                    goto doublebreak;
+                }
+                else{
+                    state->x += state->adv[ch];
+                }
+            }
+        }
+        still_looping = buffer_stringify_next(&state->stream);
+    }while(still_looping);
+    doublebreak:;
+    
+    state->i = i;
+    
+    b32 consume_token = 0;
+    if (i >= state->token_ptr->start + state->token_ptr->size){
+        consume_token = 1;
+    }
+    
+    if (consume_token){
+        switch (state->token_ptr->type){
+            case CPP_TOKEN_BRACE_OPEN:
+            {
+                state->paren_nesting[state->paren_safe_top] += state->tab_indent_amount;
+            }break;
+            
+            case CPP_TOKEN_BRACE_CLOSE:
+            {
+                state->paren_nesting[state->paren_safe_top] -= state->tab_indent_amount;
+            }break;
+            
+            case CPP_TOKEN_PARENTHESE_OPEN:
+            {
+                ++state->paren_top;
+                
+                i32 top = state->paren_top;
+                if (top >= ArrayCount(state->paren_nesting)){
+                    top = ArrayCount(state->paren_nesting) - 1;
+                }
+                state->paren_safe_top = top;
+                
+                state->paren_nesting[top] = state->x;
+            }break;
+            
+            case CPP_TOKEN_PARENTHESE_CLOSE:
+            {
+                --state->paren_top;
+                
+                if (state->paren_top < 0){
+                    state->paren_top = 0;
+                }
+                
+                i32 top = state->paren_top;
+                if (top >= ArrayCount(state->paren_nesting)){
+                    top = ArrayCount(state->paren_nesting) - 1;
+                }
+                state->paren_safe_top = top;
+            }break;
+        }
+        
+        ++state->token_ptr;
+    }
+    
+    result.position_end = state->i;
+    result.final_x = state->x;
+    
+    return(result);
+}
+
 internal void
 file_measure_wraps(Models *models, Editing_File *file, f32 font_height, f32 *adv){
-    file_allocate_wraps_as_needed(&models->mem.general, file);
+    General_Memory *general = &models->mem.general;
-    file_allocate_indents_as_needed(&models->mem.general, file, file->state.buffer.line_count);
+    file_allocate_wraps_as_needed(general, file);
-    file_allocate_wrap_positions_as_needed(&models->mem.general, file, file->state.buffer.line_count);
+    file_allocate_indents_as_needed(general, file, file->state.buffer.line_count);
+    file_allocate_wrap_positions_as_needed(general, file, file->state.buffer.line_count);
     
     Buffer_Measure_Wrap_Params params;
     params.buffer          = &file->state.buffer;
     params.wrap_line_index = file->state.wrap_line_index;
     params.adv             = adv;
     params.width           = (f32)file->settings.display_width;
-    params.virtual_white   = VWHITE;
+    params.virtual_white   = file->settings.virtual_white;
     
     i32 size = buffer_size(params.buffer);
     
@@ -1030,19 +1206,12 @@ file_measure_wraps(Models *models, Editing_File *file, f32 font_height, f32 *adv
     i32 wrap_position_index = 0;
     file->state.wrap_positions[wrap_position_index++] = 0;
     
-    Cpp_Token_Array token_array = {0};
+    Code_Wrap_State wrap_state = {0};
-    Cpp_Token *token_ptr = 0;
-    Cpp_Token *end_token = 0;
-    
-    f32 tab_indent_amount = adv['\t'];
-    f32 code_indent_level = 0;
     
     b32 use_tokens = 0;
     
     if (file->state.tokens_complete && !file->state.still_lexing){
-        token_array = file->state.token_array;
+        wrap_state_init(&wrap_state, file, adv);
-        token_ptr = token_array.tokens;
-        end_token = token_ptr + token_array.count;
         use_tokens = 1;
     }
     
@@ -1052,55 +1221,75 @@ file_measure_wraps(Models *models, Editing_File *file, f32 font_height, f32 *adv
         switch (stop.status){
             case BLStatus_NeedWrapDetermination:
             {
-                Buffer_Stream_Type stream = {0};
+                if (use_tokens){
-                
+                    Code_Wrap_Step step = wrap_state_consume_token(&wrap_state, -1);
-                i32 stage = 0;
+                    
-                i32 i = stop.pos;
+                    wrap_unit_end = step.position_end;
-                f32 x = stop.x;
+                    
-                f32 self_x = 0;
+                    if (wrap_unit_end < wrap_state.token_ptr->start){
-                if (buffer_stringify_loop(&stream, params.buffer, i, size)){
+                        wrap_unit_end = wrap_state.token_ptr->start;
-                    b32 still_looping = 0;
+                    }
-                    do{
+                    
-                        for (; i < stream.end; ++i){
+                    if (step.final_x > params.width){
-                            char ch = stream.data[i];
+                        do_wrap = 1;
-                            
+                        file_allocate_wrap_positions_as_needed(general, file, wrap_position_index);
-                            switch (stage){
+                        file->state.wrap_positions[wrap_position_index++] = stop.pos;
-                                case 0:
+                    }
-                                {
+                    else{
-                                    if (char_is_whitespace(ch)){
+                        do_wrap = 0;
-                                        stage = 1;
+                    }
-                                    }
-                                    else{
-                                        f32 adv = params.adv[ch];
-                                        x += adv;
-                                        self_x += adv;
-                                        if (self_x > params.width){
-                                            goto doublebreak;
-                                        }
-                                    }
-                                }break;
-                                
-                                case 1:
-                                {
-                                    if (!char_is_whitespace(ch)){
-                                        goto doublebreak;
-                                    }
-                                }break;
-                            }
-                        }
-                        still_looping = buffer_stringify_next(&stream);
-                    }while(still_looping);
-                }
-                
-                doublebreak:;
-                wrap_unit_end = i;
-                if (x > params.width){
-                    do_wrap = 1;
-                    file_allocate_wrap_positions_as_needed(&models->mem.general, file, wrap_position_index);
-                    file->state.wrap_positions[wrap_position_index++] = stop.pos;
                 }
                 else{
-                    do_wrap = 0;
+                    Buffer_Stream_Type stream = {0};
+                    
+                    i32 stage = 0;
+                    i32 i = stop.pos;
+                    f32 x = stop.x;
+                    f32 self_x = 0;
+                    if (buffer_stringify_loop(&stream, params.buffer, i, size)){
+                        b32 still_looping = 0;
+                        do{
+                            for (; i < stream.end; ++i){
+                                char ch = stream.data[i];
+                                
+                                switch (stage){
+                                    case 0:
+                                    {
+                                        if (char_is_whitespace(ch)){
+                                            stage = 1;
+                                        }
+                                        else{
+                                            f32 adv = params.adv[ch];
+                                            x += adv;
+                                            self_x += adv;
+                                            if (self_x > params.width){
+                                                goto doublebreak;
+                                            }
+                                        }
+                                    }break;
+                                    
+                                    case 1:
+                                    {
+                                        if (!char_is_whitespace(ch)){
+                                            goto doublebreak;
+                                        }
+                                    }break;
+                                }
+                            }
+                            still_looping = buffer_stringify_next(&stream);
+                        }while(still_looping);
+                    }
+                    
+                    doublebreak:;
+                    wrap_unit_end = i;
+                    if (x > params.width){
+                        do_wrap = 1;
+                        file_allocate_wrap_positions_as_needed(general, file, wrap_position_index);
+                        file->state.wrap_positions[wrap_position_index++] = stop.pos;
+                    }
+                    else{
+                        do_wrap = 0;
+                    }
                 }
             }break;
             
@@ -1108,41 +1297,33 @@ file_measure_wraps(Models *models, Editing_File *file, f32 font_height, f32 *adv
             case BLStatus_NeedLineShift:
             {
                 if (use_tokens){
-                    for (; token_ptr < end_token; ++token_ptr){
+                    for (; wrap_state.token_ptr < wrap_state.end_token; ){
-                        if (stop.pos < token_ptr->start + token_ptr->size){
+                        Code_Wrap_Step step = wrap_state_consume_token(&wrap_state, stop.pos);
+                        
+                        if (step.position_end >= stop.pos){
                             break;
                         }
-                        
+                    }
-                        switch (token_ptr->type){
+                    
-                            case CPP_TOKEN_BRACE_OPEN:
+                    i32 next_line_start = params.buffer->line_starts[stop.line_index+1];
-                            {
+                    f32 current_shift = wrap_state.paren_nesting[wrap_state.paren_safe_top];
-                                code_indent_level += tab_indent_amount;
+                    if (wrap_state.token_ptr->start < next_line_start){
-                            }break;
+                        if (wrap_state.token_ptr->flags & CPP_TFLAG_PP_DIRECTIVE){
-                            
+                            current_shift = 0;
-                            case CPP_TOKEN_BRACE_CLOSE:
+                        }
-                            {
+                        else{
-                                code_indent_level -= tab_indent_amount;
+                            switch (wrap_state.token_ptr->type){
-                            }break;
+                                case CPP_TOKEN_BRACE_CLOSE:
+                                {
+                                    if (wrap_state.paren_safe_top == 0){
+                                        current_shift -= wrap_state.tab_indent_amount;
+                                    }
+                                }break;
+                            }
                         }
                     }
                     
-                    // TODO(allen): Make sure we can put the "size" at the end of the line count array.
+                    line_shift = current_shift;
-                    i32 next_line_start = size;
-                    if (stop.line_index < params.buffer->line_count){
-                        params.buffer->line_starts[stop.line_index+1];
-                    }
-                    
-                    f32 immediate_shift = 0;
-                    if (token_ptr->start < next_line_start){
-                        switch (token_ptr->type){
-                            case CPP_TOKEN_BRACE_CLOSE:
-                            {
-                                immediate_shift -= tab_indent_amount;
-                            }break;
-                        }
-                    }
-                    
-                    line_shift = code_indent_level + immediate_shift;
                     if (line_shift < 0){
                         line_shift = 0;
                     }
@@ -1156,18 +1337,20 @@ file_measure_wraps(Models *models, Editing_File *file, f32 font_height, f32 *adv
                 }
                 
                 if (stop.wrap_line_index >= file->state.line_indent_max){
-                    file_allocate_indents_as_needed(&models->mem.general, file, stop.wrap_line_index);
+                    file_allocate_indents_as_needed(general, file, stop.wrap_line_index);
                 }
                 
                 file->state.line_indents[stop.wrap_line_index] = line_shift;
                 file->state.wrap_line_count = stop.wrap_line_index;
+                
+                wrap_state_set_x(&wrap_state, line_shift);
             }break;
         }
     }while(stop.status != BLStatus_Finished);
     
     ++file->state.wrap_line_count;
     
-    file_allocate_wrap_positions_as_needed(&models->mem.general, file, wrap_position_index);
+    file_allocate_wrap_positions_as_needed(general, file, wrap_position_index);
     file->state.wrap_positions[wrap_position_index++] = size;
     file->state.wrap_position_count = wrap_position_index;
 }
@@ -2287,7 +2470,7 @@ file_do_single_edit(System_Functions *system,
     
     file_allocate_character_starts_as_needed(general, file);
     buffer_remeasure_character_starts(buffer, line_start, line_end, line_shift,
-                                      file->state.character_starts, 0, VWHITE);
+                                      file->state.character_starts, 0, file->settings.virtual_white);
     
     // TODO(allen): Redo this as some sort of dialectic API
 #if 0
@@ -2403,7 +2586,7 @@ file_do_batch_edit(System_Functions *system, Models *models, Editing_File *file,
     
     // TODO(allen): write the remeasurement version
     file_allocate_character_starts_as_needed(&models->mem.general, file);
-    buffer_measure_character_starts(&file->state.buffer, file->state.character_starts, 0, VWHITE);
+    buffer_measure_character_starts(&file->state.buffer, file->state.character_starts, 0, file->settings.virtual_white);
     
     Render_Font *font = get_font_info(models->font_set, file->settings.font_id)->font;
     file_measure_wraps(models, file, (f32)font->height, font->advance_data);
@@ -5085,7 +5268,7 @@ draw_file_loaded(View *view, i32_Rect rect, b32 is_active, Render_Target *target
         params.wrapped       = wrapped;
         params.font_height   = (f32)line_height;
         params.adv           = advance_data;
-        params.virtual_white = VWHITE;
+        params.virtual_white = file->settings.virtual_white;
         
         Buffer_Render_State state = {0};
         Buffer_Layout_Stop stop = {0};
@@ -5176,18 +5359,18 @@ draw_file_loaded(View *view, i32_Rect rect, b32 is_active, Render_Target *target
             
             if (token_i < token_array.count){
                 if (ind >= token_array.tokens[token_i].start){
-                    main_color =
+                    while (ind >= token_array.tokens[token_i].start){
-                        *style_get_color(style, token_array.tokens[token_i]);
+                        main_color = *style_get_color(style, token_array.tokens[token_i]);
-                    current_token = token_array.tokens[token_i];
+                        current_token = token_array.tokens[token_i];
-                    ++token_i;
+                        ++token_i;
+                    }
                 }
                 else if (ind >= current_token.start + current_token.size){
                     main_color = style->main.default_color;
                 }
             }
             
-            if (current_token.type == CPP_TOKEN_JUNK &&
+            if (current_token.type == CPP_TOKEN_JUNK && ind >= current_token.start && ind < current_token.start + current_token.size){
-                ind >= current_token.start && ind < current_token.start + current_token.size){
                 highlight_color = style->main.highlight_junk_color;
             }
             else{
@@ -5196,10 +5379,11 @@ draw_file_loaded(View *view, i32_Rect rect, b32 is_active, Render_Target *target
         }
         
         u32 char_color = main_color;
-        if (item->flags & BRFlag_Special_Character) char_color = special_color;
+        if (item->flags & BRFlag_Special_Character){
+            char_color = special_color;
+        }
         
-        f32_Rect char_rect = f32R(item->x0, item->y0,
+        f32_Rect char_rect = f32R(item->x0, item->y0, item->x1,  item->y1);
-                                  item->x1,  item->y1);
         
         if (view->file_data.show_whitespace && highlight_color == 0 &&
             char_is_whitespace((char)item->glyphid)){

TODO.txt

@@ -157,18 +157,28 @@
 ; [X] tab to complete folder names in the new file dialogue
 
 ; Arbitrary wrap positions
-;     [X] allow for arbitrary wrap positions independent of view width
+; [X] allow for arbitrary wrap positions independent of view width
-;     [X] command for adjusting wrap positions in views
+; [X] command for adjusting wrap positions in views
-;     [X] get horizontal scrolling to work in line wrap mode
+; [X] get horizontal scrolling to work in line wrap mode
-;     [] command for setting wrap positions in views directly
+; [X] word level wrapping
-;     [] ability to see the wrap position as a number/line and adjust graphically
+; [X] ability to toggle virtual white space
-;     [] word level wrapping
+; [] command for setting wrap positions in views directly
-;     [] code level wrapping
+; [] ability to see the wrap position as a number/line and adjust graphically
+; [] home/end should go to the beginning/end of the visual line not the textual line
+; [] code level wrapping
+;     [X] handle basic scope indents
+;     [X] handle parentheses
+;     [X] handle preprocessor
+;     [X] unify wrap rule parsing and indent rule parsing
+;     [] smarter wrap rule
+;     [] handle unclosed statements
+;     [] handle comments
+;     [] fix issues when relexing happens in parallel
+; [] remeasure version of measure_wraps
 
 ; Buffer behavior cleanup
 ;    [X] show all characters as \# if they can't be rendered
 ;    [X] get the navigation working correctly around multi-glyph characters
-;    [] wrap remeasuring routine for local edits
 ;    [] provide full cursor positioning in buffers
 ;    [] provide cursor positioning system that does columns correctly
 ;    [] binary buffer mode

buffer/4coder_buffer_abstract.cpp

@@ -309,10 +309,10 @@ buffer_measure_wrap_y(Buffer_Measure_Wrap_State *S_ptr, Buffer_Measure_Wrap_Para
                                 ++S.current_wrap_index;
                                 
                                 if (params.virtual_white){
-                                    S_stop.status = BLStatus_NeedWrapLineShift;
+                                    S_stop.status          = BLStatus_NeedWrapLineShift;
-                                    S_stop.line_index = S.line_index - 1;
+                                    S_stop.line_index      = S.line_index - 1;
                                     S_stop.wrap_line_index = S.current_wrap_index;
-                                    S_stop.pos = S.i+1;
+                                    S_stop.pos             = S.i;
                                     DrYield(3, S_stop);
                                 }