Change interface naturalize function #6486

@mgoetze5 commented on GitHub (May 17, 2022):

While someone is looking at this... on NetApp an "e" is prepended to Ethernet interfaces whereas FC interfaces have nothing prepended. The current sort looks like this:

10a
10b
10c
10d
e9a
e9e
e11a
e11e

but I would prefer:

e9a
e9e
10a
10b
10c
10d
e11a
e11e

@mgoetze5 commented on GitHub (May 17, 2022): While someone is looking at this... on NetApp an "e" is prepended to Ethernet interfaces whereas FC interfaces have nothing prepended. The current sort looks like this: ``` 10a 10b 10c 10d e9a e9e e11a e11e ``` but I would prefer: ``` e9a e9e 10a 10b 10c 10d e11a e11e ```

adam commented

@sleepinggenius2 commented on GitHub (May 18, 2022):

I have the same issue in two places, but would also need to add '-' to the list of split characters.

On Cisco ASR 9K 1/10G linecards you can specify which interfaces are 1G and which are 10G and the router accordingly labels them as GigabitEthernet and TenGigE, but they currently end up sorted with all the 1G interfaces first, then the 10G ones, and not in proper port order, which is really confusing. With modules now, even putting them all on the same module doesn't fix the sort order.

The other example is on a Cisco ONS 15454/NCS 2K you can have a QSFP+ 4x10G breakout optic inserted that will create something like VCFAC-1-1-1 for the optic and VLINE-1-1-1-1-[1-4] for the individual 10G lanes, except it sorts it so that all the facility interfaces come first, then the line interfaces. In that case, they can even be modeled as child interfaces and that doesn't affect the sort order either.

@sleepinggenius2 commented on GitHub (May 18, 2022): I have the same issue in two places, but would also need to add `'-'` to the list of split characters. On Cisco ASR 9K 1/10G linecards you can specify which interfaces are 1G and which are 10G and the router accordingly labels them as GigabitEthernet and TenGigE, but they currently end up sorted with all the 1G interfaces first, then the 10G ones, and not in proper port order, which is really confusing. With modules now, even putting them all on the same module doesn't fix the sort order. The other example is on a Cisco ONS 15454/NCS 2K you can have a QSFP+ 4x10G breakout optic inserted that will create something like VCFAC-1-1-1 for the optic and VLINE-1-1-1-1-[1-4] for the individual 10G lanes, except it sorts it so that all the facility interfaces come first, then the line interfaces. In that case, they can even be modeled as child interfaces and that doesn't affect the sort order either.

adam commented

@PieterL75 commented on GitHub (May 19, 2022):

While someone is looking at this... on NetApp an "e" is prepended to Ethernet interfaces whereas FC interfaces have nothing prepended. The current sort looks like this:
10a
10b
10c
10d
e9a
e9e
e11a
e11e
but I would prefer:
e9a
e9e
10a
10b
10c
10d
e11a
e11e

That will be difficult. the order function only knows the interface names, not the underlaying device model.

@PieterL75 commented on GitHub (May 19, 2022): > While someone is looking at this... on NetApp an "e" is prepended to Ethernet interfaces whereas FC interfaces have nothing prepended. The current sort looks like this: > > ``` > 10a > 10b > 10c > 10d > e9a > e9e > e11a > e11e > ``` > > but I would prefer: > > ``` > e9a > e9e > 10a > 10b > 10c > 10d > e11a > e11e > ``` That will be difficult. the order function only knows the interface names, not the underlaying device model.

adam commented

@DanSheps commented on GitHub (May 19, 2022):

I think there could be some room for improvement, perhaps we have to look at considering how the sort happens when a slot, module, or port are missing.

That said, I think we still need to maintain the sort with the interface type, but perhaps have interface type be sorted just before port (if it isn't already).

Maybe it would make sense for all of these to be individual fields and then have the name be a computed field that is the combination of all fields.

This sort of individual field setup would only work well with most network switches, regular interfaces for the most part, don't have a concept of slot/module.

@DanSheps commented on GitHub (May 19, 2022): I think there could be some room for improvement, perhaps we have to look at considering how the sort happens when a slot, module, or port are missing. That said, I think we still need to maintain the sort with the interface type, but perhaps have interface type be sorted just before port (if it isn't already). Maybe it would make sense for all of these to be individual fields and then have the name be a computed field that is the combination of all fields. This sort of individual field setup would only work well with most network switches, regular interfaces for the most part, don't have a concept of slot/module.

adam commented

@PieterL75 commented on GitHub (May 22, 2022):

Dan, why would we need to name the parts of an interface and try to fit ?
I think that is we just make the numbers a zero pretended 5 digit number, that all sorting cases work. (Except the one mgoetze5 mentioned).

@PieterL75 commented on GitHub (May 22, 2022): Dan, why would we need to name the parts of an interface and try to fit ? I think that is we just make the numbers a zero pretended 5 digit number, that all sorting cases work. (Except the one mgoetze5 mentioned).

adam commented

@jeremystretch commented on GitHub (May 24, 2022):

@PieterL75 you can find the existing tests for the natural ordering of interfaces here. As it has taken several years for us as a community to settle on the current scheme, we will only entertain modifications that do not alter or violate any of the existing tests. Is this something you want to pursue?

@jeremystretch commented on GitHub (May 24, 2022): @PieterL75 you can find the existing tests for the natural ordering of interfaces [here](https://github.com/netbox-community/netbox/blob/develop/netbox/dcim/tests/test_natural_ordering.py). As it has taken several years for us as a community to settle on the current scheme, we will only entertain modifications that do not alter or violate any of the existing tests. Is this something you want to pursue?

adam commented

@PieterL75 commented on GitHub (May 24, 2022):

So, as long as the new sorting returns the same order for the test , we should be good?
The 'calculated' values will change of course, the displayed order must remain the same.

@PieterL75 commented on GitHub (May 24, 2022): So, as long as the new sorting returns the same order for the test , we should be good? The 'calculated' values will change of course, the displayed order must remain the same.

adam commented

@jeremystretch commented on GitHub (May 24, 2022):

So, as long as the new sorting returns the same order for the test , we should be good?

Provided the proposed change makes sense, yes. It'll also need a new test for the scenario you described above.

@jeremystretch commented on GitHub (May 24, 2022): > So, as long as the new sorting returns the same order for the test , we should be good? Provided the proposed change makes sense, yes. It'll also need a new test for the scenario you described above.

adam commented

@jeremystretch commented on GitHub (May 24, 2022):

The 'calculated' values will change of course, the displayed order must remain the same.

We need to be careful with this. Any changes that would result in recalculating raw names will require that users run renaturalize against all interfaces when upgrading, which is best avoided.

@jeremystretch commented on GitHub (May 24, 2022): > The 'calculated' values will change of course, the displayed order must remain the same. We need to be careful with this. Any changes that would result in recalculating raw names will require that users run `renaturalize` against _all_ interfaces when upgrading, which is best avoided.

adam commented

@PieterL75 commented on GitHub (May 24, 2022):

Sounds good. Im good to create a PR for this

@PieterL75 commented on GitHub (May 24, 2022): Sounds good. Im good to create a PR for this

adam commented

@PieterL75 commented on GitHub (May 24, 2022):

I've been thinking that too, but it might be something that is inevitable and needs to be weighted...
Could (if needed) a migration script take care of that?

@PieterL75 commented on GitHub (May 24, 2022): I've been thinking that too, but it might be something that is inevitable and needs to be weighted... Could (if needed) a migration script take care of that?

adam commented

@DanSheps commented on GitHub (May 24, 2022):

Looking at the code, I think this is because it treats empty slot, subslot, position, subposition as "9999", would it make sense to treat these as "0000" instead? That would likely solve this specific use case.

@DanSheps commented on GitHub (May 24, 2022): Looking at the code, I think this is because it treats empty slot, subslot, position, subposition as "9999", would it make sense to treat these as "0000" instead? That would likely solve this specific use case.

adam commented

@PieterL75 commented on GitHub (May 24, 2022):

I see now in that test script why the interface type is placed after the port's..

            'GigabitEthernet0/1',
            'GigabitEthernet0/2',
            'GigabitEthernet0/10',
            'TenGigabitEthernet0/20',
            'TenGigabitEthernet0/21',

same for junipers where xe, ge and te interfaces can be mixed in one 'module'

We dont want those TenG before the Gig's
Going to think some more, and check @DanSheps advise

@PieterL75 commented on GitHub (May 24, 2022): I see now in that test script why the interface type is placed after the port's.. ``` 'GigabitEthernet0/1', 'GigabitEthernet0/2', 'GigabitEthernet0/10', 'TenGigabitEthernet0/20', 'TenGigabitEthernet0/21', ``` same for junipers where xe, ge and te interfaces can be mixed in one 'module' We dont want those TenG before the Gig's Going to think some more, and check @DanSheps advise

adam commented

@PieterL75 commented on GitHub (Jun 1, 2022):

@DanSheps @jeremystretch
I have a version of the naturalize_interface ready in my fork.
Shall I create a PR to have it reviewed ? or will you first check it out.
https://github.com/PieterL75/netbox/tree/issue_9368

It does change the rawnames, as it uses a totally different logic.
There is an option to change the logic, based on the name of the interface, and the model properties..
The current implementation passes the ./manage.py test dcim.tests.test_natural_ordering
and it also sorts the ones in the comments (netapp, vmac, ....)

@PieterL75 commented on GitHub (Jun 1, 2022): @DanSheps @jeremystretch I have a version of the naturalize_interface ready in my fork. Shall I create a PR to have it reviewed ? or will you first check it out. https://github.com/PieterL75/netbox/tree/issue_9368 It does change the rawnames, as it uses a totally different logic. There is an option to change the logic, based on the name of the interface, and the model properties.. The current implementation passes the ```./manage.py test dcim.tests.test_natural_ordering``` and it also sorts the ones in the comments (netapp, vmac, ....)

adam commented

@DanSheps commented on GitHub (Jun 1, 2022):

Sure, push up a PR and we will take a look

@DanSheps commented on GitHub (Jun 1, 2022): Sure, push up a PR and we will take a look

adam commented

@PieterL75 commented on GitHub (Jun 7, 2022):

I have to fix the tests so that they match the new naturalization function....
But, what I see that the biggest 'exception' is that causes my biggest headache :
The sorting requirement that 'FastEthernet0' comes after 'GigaEthernet 0/1'.
I don't see a reason to do that, except that the FastEthernet0 is a mgmt only interface that you don't want 'sorted' with the rest.
For that I added a condition that places the mgmt_only interface always at the end of the sorting.
If that adheres the requirements of why Fa0 was after Gi0/1, then I would like to get rid of that hard requirement. (I dont have the history thread on how the current function was born)

@PieterL75 commented on GitHub (Jun 7, 2022): I have to fix the tests so that they match the new naturalization function.... But, what I see that the biggest 'exception' is that causes my biggest headache : The sorting requirement that 'FastEthernet0' comes after 'GigaEthernet 0/1'. I don't see a reason to do that, except that the FastEthernet0 is a mgmt only interface that you don't want 'sorted' with the rest. For that I added a condition that places the mgmt_only interface always at the end of the sorting. If that adheres the requirements of why Fa0 was after Gi0/1, then I would like to get rid of that hard requirement. (I dont have the history thread on how the current function was born)

adam commented

@DanSheps commented on GitHub (Jun 10, 2022):

I don't see a reason to do that, except that the FastEthernet0 is a mgmt only interface that you don't want 'sorted' with the rest.
For that I added a condition that places the mgmt_only interface always at the end of the sorting.

I will take a look at the tests and see if I can figure out why this is. To me, FA should come before GI, especially because "0" is the port. I believe this is because in the naturalize we use 9999 for slot and module when it is non-existant, but to me that is backwards and it should be 0000 for a null value.

@DanSheps commented on GitHub (Jun 10, 2022): > I don't see a reason to do that, except that the FastEthernet0 is a mgmt only interface that you don't want 'sorted' with the rest. > For that I added a condition that places the mgmt_only interface always at the end of the sorting. I will take a look at the tests and see if I can figure out why this is. To me, FA should come before GI, especially because "0" is the port. I believe this is because in the naturalize we use 9999 for slot and module when it is non-existant, but to me that is backwards and it should be 0000 for a null value.

adam commented

@candlerb commented on GitHub (Jun 23, 2022):

I suggested the same sort of general natural ordering algorithm at https://github.com/netbox-community/netbox/issues/6882#issuecomment-898508667 and https://github.com/netbox-community/netbox/issues/2872#issuecomment-491279206 - rejected both times.

Those links include some discussion of how this doesn't agree with the existing test cases.

@candlerb commented on GitHub (Jun 23, 2022): I suggested the same sort of general natural ordering algorithm at https://github.com/netbox-community/netbox/issues/6882#issuecomment-898508667 and https://github.com/netbox-community/netbox/issues/2872#issuecomment-491279206 - rejected both times. Those links include some discussion of how this doesn't agree with the existing test cases.

adam commented

@PieterL75 commented on GitHub (Jun 25, 2022):

To me, it seems that the decisions made in the past were not the best.
We should start a new discussion to agree on a new sorting algorithm that will work for a lot of cases.
The corner cases that are now taken into account, and mess up other natural sorting, should go away in favor of a 'logical' scheme without exceptions. (ex : placing FastEthernet1 at the end, because on some switch types that is the mgmt interface).

I'll work out a generic sorting algorithm and post a new PR to test. This will ofcourse change the current values of _name. But that can be handled with a migration schema file. Rollback is not possible, but that is simply not possible in an version of netbox. Forward is the only way :-)

@PieterL75 commented on GitHub (Jun 25, 2022): To me, it seems that the decisions made in the past were not the best. We should start a new discussion to agree on a new sorting algorithm that will work for a lot of cases. The corner cases that are now taken into account, and mess up other natural sorting, should go away in favor of a 'logical' scheme without exceptions. (ex : placing FastEthernet1 at the end, because on some switch types that is the mgmt interface). I'll work out a generic sorting algorithm and post a new PR to test. This will ofcourse change the current values of _name. But that can be handled with a migration schema file. Rollback is not possible, but that is simply not possible in an version of netbox. Forward is the only way :-)

adam commented

@DanSheps commented on GitHub (Jul 2, 2022):

@PieterL75 if you want to post up the new FR we can definitely discuss modifying the ordering.

@DanSheps commented on GitHub (Jul 2, 2022): @PieterL75 if you want to post up the new FR we can definitely discuss modifying the ordering.

adam commented

@PieterL75 commented on GitHub (Jul 4, 2022):

@DanSheps I pushed a new PR https://github.com/netbox-community/netbox/pull/9463
I suspect it will not be approved yet, but I hope I was able to build a logic that is easily extendible.

@PieterL75 commented on GitHub (Jul 4, 2022): @DanSheps I pushed a new PR https://github.com/netbox-community/netbox/pull/9463 I suspect it will not be approved yet, but I hope I was able to build a logic that is easily extendible.

adam commented

@PieterL75 commented on GitHub (Jul 5, 2022):

I had a very interesting discussion.. @jeremystretch @DanSheps
Why bother with the naturalize function ?
If we could manually set/change/dragdrop the order on the device/module-type, then that order can be taken for the derived devices.
That sort-order can be retained on a device. if a new interface is added to the device itself, then it could be placed at any place and stored....
the _name field can then simply become 0001, 0002, 0003, ...

@PieterL75 commented on GitHub (Jul 5, 2022): I had a very interesting discussion.. @jeremystretch @DanSheps Why bother with the naturalize function ? If we could manually set/change/dragdrop the order on the device/module-type, then that order can be taken for the derived devices. That sort-order can be retained on a device. if a new interface is added to the device itself, then it could be placed at any place and stored.... the _name field can then simply become 0001, 0002, 0003, ...

adam commented

@PieterL75 commented on GitHub (Jul 8, 2022):

@jeremystretch @candlerb did you had time to review the pr I submitted?
However, I feel more for solution that interfaces are ordered manually in the template, or on the device... No more debates, each model sorts them by it's design

@PieterL75 commented on GitHub (Jul 8, 2022): @jeremystretch @candlerb did you had time to review the pr I submitted? However, I feel more for solution that interfaces are ordered manually in the template, or on the device... No more debates, each model sorts them by it's design

adam commented

@DanSheps commented on GitHub (Jul 9, 2022):

I don't think a manual sort is going to be in the cards, it is too difficult to manage. We had it, pulled it out for this.

That said, I think a better method could be devised.

@DanSheps commented on GitHub (Jul 9, 2022): I don't think a manual sort is going to be in the cards, it is too difficult to manage. We had it, pulled it out for this. That said, I think a better method could be devised.

adam commented

@PieterL75 commented on GitHub (Jul 9, 2022):

Did you had it when templates existed? Because with those, it is just a one time manual sort.
An the option to change it a bit when it became a device...
Modules might make it a challenge though..

@PieterL75 commented on GitHub (Jul 9, 2022): Did you had it when templates existed? Because with those, it is just a one time manual sort. An the option to change it a bit when it became a device... Modules might make it a challenge though..

adam commented

@PieterL75 commented on GitHub (Aug 27, 2022):

@jeremystretch
On the PR you mention that you will not allow the current naturalization function to be changed.
As this current one is really not doing what it should for multiple devicetypes, I have some suggestions.

custom naturalization function, either by a plugin or by a config setting
manual ordering by using templates.

I prefer the manual ordering, as it will cover all possible situations. The initial ordering can be fine by such a function, but manual reordering should be an option.

@PieterL75 commented on GitHub (Aug 27, 2022): @jeremystretch On the PR you mention that you will not allow the current naturalization function to be changed. As this current one is really not doing what it should for multiple devicetypes, I have some suggestions. - custom naturalization function, either by a plugin or by a config setting - manual ordering by using templates. I prefer the manual ordering, as it will cover all possible situations. The initial ordering can be fine by such a function, but manual reordering should be an option.

adam commented

@PieterL75 commented on GitHub (Sep 26, 2022):

Still wondering on how this can be solved.
'One Solution Fits All' will not work, so we have to be able to provide a way to customize the naturalization.

Either by a plugin that can overrule the default, or by a configuration.py setting that refers to a py function.
Or a manual reorder..

@PieterL75 commented on GitHub (Sep 26, 2022): Still wondering on how this can be solved. 'One Solution Fits All' will not work, so we have to be able to provide a way to customize the naturalization. Either by a plugin that can overrule the default, or by a configuration.py setting that refers to a py function. Or a manual reorder..

adam commented

@ZPrimed commented on GitHub (Sep 30, 2022):

Would it be possible to get the GUI do a multi-column sort? I.e. sort by column A first, then sort by column B? I know this is something that exists in some operating systems and programs, but I don't know how feasible it is to implement within Django...

This could potentially be very helpful, as a user could sort by Interface Type first, and then by Name, which would effectively allow you to "force" (or at least influence) the grouping based on the port speed.

Plus, on many devices, the physical port labels are just a consecutive numeric string, so it would open the possibility to simply sort by "Label" and get everything in the "physical port order". (There currently seems to be a bug in the sorting for the "Label" column - 1, 10, and 11 are next to each other... I was searching to see if it had been reported when I turned up this discussion!)

@ZPrimed commented on GitHub (Sep 30, 2022): Would it be possible to get the GUI do a multi-column sort? I.e. sort by column A first, then sort by column B? I know this is something that exists in some operating systems and programs, but I don't know how feasible it is to implement within Django... This could potentially be very helpful, as a user could sort by Interface Type first, and then by Name, which would effectively allow you to "force" (or at least influence) the grouping based on the port speed. Plus, on many devices, the physical port labels are just a consecutive numeric string, so it would open the possibility to simply sort by "Label" and get everything in the "physical port order". (There currently seems to be a bug in the sorting for the "Label" column - 1, 10, and 11 are next to each other... I was searching to see if it had been reported when I turned up this discussion!)

adam commented