CSV-based bulk update functionality #5721

@DanSheps commented on GitHub (Dec 2, 2021):

I think a better way to phrase this would be "bulk update via import"

@DanSheps commented on GitHub (Dec 2, 2021): I think a better way to phrase this would be "bulk update via import"

adam commented

@bhunt64848 commented on GitHub (Dec 2, 2021):

Agreed!

@bhunt64848 commented on GitHub (Dec 2, 2021): Agreed!

adam commented

@Mizarv commented on GitHub (Dec 6, 2021):

This has already been suggsted in #1732 with a reply from Jeremy.

Given it was 4 years ago, I would love to see this looked at again.

@Mizarv commented on GitHub (Dec 6, 2021): This has already been suggsted in #1732 with a reply from Jeremy. Given it was 4 years ago, I would love to see this looked at again.

adam commented

@jeremystretch commented on GitHub (Dec 8, 2021):

I'm afraid this needs quite a bit more detail. What is the proposed workflow? How will you accurately identify existing objects?

@jeremystretch commented on GitHub (Dec 8, 2021): I'm afraid this needs quite a bit more detail. What is the proposed workflow? How will you accurately identify existing objects?

adam commented

@Mizarv commented on GitHub (Dec 9, 2021):

I guess now that we can export the CSVs with the object ID you could use that as that field can't be updated.

Maybe an additional tab to avoid complicating the existing CSV Data method as this works excellently

@Mizarv commented on GitHub (Dec 9, 2021): I guess now that we can export the CSVs with the object ID you could use that as that field can't be updated. Maybe an additional tab to avoid complicating the existing CSV Data method as this works excellently ![image](https://user-images.githubusercontent.com/91476976/145370016-ff497871-08f7-43ea-9a49-c465eb2f8ca3.png)

adam commented

@martinum4 commented on GitHub (Dec 10, 2021):

Maybe it could the same as the normal csv import in regard to attribute extension and selection, example:

device.id,serial,asset_tag 
500,0815,47111234

would be used to update the serial and asset tags for the device with ID 500.
However it could also be like

device,serial,asset_tag 
TestDevice,0815,47111234

since the name is an unique identifier too.

The next page could be similar to the interface-renaming screen where the prechange and postchange-data for each device is displayed, that way it could be avoided to update already set fields.

@martinum4 commented on GitHub (Dec 10, 2021): Maybe it could the same as the normal csv import in regard to attribute extension and selection, example: ``` device.id,serial,asset_tag 500,0815,47111234 ``` would be used to update the serial and asset tags for the device with ID 500. However it could also be like ``` device,serial,asset_tag TestDevice,0815,47111234 ``` since the name is an unique identifier too. The next page could be similar to the interface-renaming screen where the prechange and postchange-data for each device is displayed, that way it could be avoided to update already set fields.

adam commented

@jeremystretch commented on GitHub (Dec 13, 2021):

Implementing this would require an absolutely reliable unique object identifier, i.e. the object's primary key (numeric ID). If that's acceptable, I think this is reasonably feasible to implement. However, it would likely require a separate view and workflow from the bulk import function, as we would be updating objects rather than creating them, and none of the form fields would be required (similar to the existing bulk edit functionality).

@jeremystretch commented on GitHub (Dec 13, 2021): Implementing this would require an absolutely reliable unique object identifier, i.e. the object's primary key (numeric ID). If that's acceptable, I think this is reasonably feasible to implement. However, it would likely require a separate view and workflow from the bulk import function, as we would be updating objects rather than creating them, and none of the form fields would be required (similar to the existing bulk edit functionality).

adam commented

@github-actions[bot] commented on GitHub (Feb 12, 2022):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. Please see our contributing guide.

@github-actions[bot] commented on GitHub (Feb 12, 2022): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. NetBox is governed by a small group of core maintainers which means not all opened issues may receive direct feedback. Please see our [contributing guide](https://github.com/netbox-community/netbox/blob/develop/CONTRIBUTING.md).

adam commented

@mikewilliamsjr commented on GitHub (Jun 8, 2022):

Would addressing this issue include ensuring that the data that is exported can be immediately imported once the data is deleted, or can be imported to a fresh instance without curation? Currently, bulk import demands headers formatted in lower-case and requires that some content is in a specific format. If data must be imported in a specific format, the data should be exported in the same format as what is acceptable for import. Usability will be improved and admin effort will be reduced.

@mikewilliamsjr commented on GitHub (Jun 8, 2022): Would addressing this issue include ensuring that the data that is exported can be immediately imported once the data is deleted, or can be imported to a fresh instance without curation? Currently, bulk import demands headers formatted in lower-case and requires that some content is in a specific format. If data must be imported in a specific format, the data should be exported in the same format as what is acceptable for import. Usability will be improved and admin effort will be reduced.

adam commented

@jeremystretch commented on GitHub (Aug 11, 2022):

Would addressing this issue include ensuring that the data that is exported can be immediately imported once the data is deleted, or can be imported to a fresh instance without curation?

No. This has been discussed many times and in many cases is not feasible due to import requiring sufficiently unique identifiers for related objects, which would be extraneous in exported data.

@jeremystretch commented on GitHub (Aug 11, 2022): > Would addressing this issue include ensuring that the data that is exported can be immediately imported once the data is deleted, or can be imported to a fresh instance without curation? No. This has been discussed many times and in many cases is not feasible due to import requiring sufficiently unique identifiers for related objects, which would be extraneous in exported data.

adam commented

@mikewilliamsjr commented on GitHub (Aug 11, 2022):

Would addressing this issue include ensuring that the data that is exported can be immediately imported once the data is deleted, or can be imported to a fresh instance without curation?

No. This has been discussed many times and in many cases is not feasible due to import requiring sufficiently unique identifiers for related objects, which would be extraneous in exported data.

You've acknowledged that including the identifiers in imported data would make martinum4's suggestion feasible. How is their suggestion materially different from mine, aside from their pure focus on importing updates? Support for updating existing records is effectively the same as importing new, unique records. The same process for ensuring primary keys are not duplicated could be used in both instances.

The logical next step is to provide a means for easily accessing the primary keys. A toggled setting to include/exclude the keys on export could improve the usability of this tool, especially if your intent is to develop a new workflow for bulk update. The keys are only extraneous if the intent is to use exported data in some other application with no intent to import; if the intent is to migrate to a fresh instance or programmatically update existing data, then rather than being extraneous, the keys seem mandatory.

@mikewilliamsjr commented on GitHub (Aug 11, 2022): > > Would addressing this issue include ensuring that the data that is exported can be immediately imported once the data is deleted, or can be imported to a fresh instance without curation? > > No. This has been discussed many times and in many cases is not feasible due to import requiring sufficiently unique identifiers for related objects, which would be extraneous in exported data. You've acknowledged that including the identifiers in imported data would make martinum4's suggestion feasible. How is their suggestion materially different from mine, aside from their pure focus on importing updates? Support for updating existing records is effectively the same as importing new, unique records. The same process for ensuring primary keys are not duplicated could be used in both instances. The logical next step is to provide a means for easily accessing the primary keys. A toggled setting to include/exclude the keys on export could improve the usability of this tool, especially if your intent is to develop a new workflow for bulk update. The keys are only extraneous if the intent is to use exported data in some other application with no intent to import; if the intent is to migrate to a fresh instance or programmatically update existing data, then rather than being extraneous, the keys seem mandatory.

adam commented

@jeremystretch commented on GitHub (Aug 11, 2022):

How is their suggestion materially different from mine, aside from their pure focus on importing updates?

A primary key don't exist until the object has been created in the database. It is not possible to reference one until after the object has been imported.

Let's please keep this discussion limited to the scope of the FR, which involves the bulk update of existing objects.

@jeremystretch commented on GitHub (Aug 11, 2022): > How is their suggestion materially different from mine, aside from their pure focus on importing updates? A primary key don't exist until the object has been created in the database. It is not possible to reference one until _after_ the object has been imported. Let's please keep this discussion limited to the scope of the FR, which involves the bulk update of existing objects.

adam commented

@mikewilliamsjr commented on GitHub (Aug 12, 2022):

A primary key don't exist until the object has been created in the database. It is not possible to reference one until after the object has been imported.

RIght; hence, the suggestion to add an option to include the key in exported data.

The point I'm making is that both requests can be supported with a more useful export. IOW, by making it possible to include the keys for each record with the exported data, you would enable users to export the data, manipulate the data as required, then either import to the existing instance (update) or to a new instance.

Let's please keep this discussion limited to the scope of the FR, which involves the bulk update of existing objects.

Please forgive me for not realizing how I'm offtopic. Unless my understanding is far off, it seems like the path to CSV-based bulk updates would include a means for retrieving the primary key of each record. From there, the next logical step seems to be to ensure that exported column headers match the required column headers for import. Each of these steps should simplify the logic and/or administrative effort required for data validation.

@mikewilliamsjr commented on GitHub (Aug 12, 2022): > A primary key don't exist until the object has been created in the database. It is not possible to reference one until _after_ the object has been imported. RIght; hence, the suggestion to add an option to include the key in exported data. The point I'm making is that both requests can be supported with a more useful export. IOW, by making it possible to include the keys for each record with the exported data, you would enable users to export the data, manipulate the data as required, then either import to the existing instance (update) or to a new instance. > Let's please keep this discussion limited to the scope of the FR, which involves the bulk update of existing objects. Please forgive me for not realizing how I'm offtopic. Unless my understanding is far off, it seems like the path to CSV-based bulk updates would include a means for retrieving the primary key of each record. From there, the next logical step seems to be to ensure that exported column headers match the required column headers for import. Each of these steps should simplify the logic and/or administrative effort required for data validation.

adam commented

@jdavidson2021 commented on GitHub (Aug 12, 2022):

Maybe have two choices during the CSV export to provide the data with "friendly names" from what is currently shown in the GUI (current design) and 2nd choice to export the data with "import column names". We currently export the data in CSV, delete the objects, massage all of the column names that won't import correctly from the current CSV export, update our data in the CSV, and then re-import the data. I'm sure this is what most users are doing when they need to do a bulk update of information currently.

@jdavidson2021 commented on GitHub (Aug 12, 2022): Maybe have two choices during the CSV export to provide the data with "friendly names" from what is currently shown in the GUI (current design) and 2nd choice to export the data with "import column names". We currently export the data in CSV, delete the objects, massage all of the column names that won't import correctly from the current CSV export, update our data in the CSV, and then re-import the data. I'm sure this is what most users are doing when they need to do a bulk update of information currently.

adam commented