Non-ascii tag creates empty slug, exception django.urls.exceptions.NoReverseMatch #3038

@DanSheps commented on GitHub (Dec 2, 2019):

This looks to be a simple missing dependancy of TagBase for "unidecode", as otherwise slugify() in TagBase just returns the "tag" without any cleanup.

Will ponder how to best deal with this.

@DanSheps commented on GitHub (Dec 2, 2019): This looks to be a simple missing dependancy of TagBase for "unidecode", as otherwise slugify() in TagBase just returns the "tag" without any cleanup. Will ponder how to best deal with this.

adam commented

@kobayashi commented on GitHub (Dec 5, 2019):

+1 just adding unidecode as a required package. That one is simplest.
An another way is override slugify described in taggit docs though, that will lead lots of work or make it complicated just for setting slug from tag name like some comments here

@kobayashi commented on GitHub (Dec 5, 2019): +1 just adding unidecode as a required package. That one is simplest. An another way is override `slugify` described in [taggit docs](https://django-taggit.readthedocs.io/en/latest/custom_tagging.html#TagBase.slugify) though, that will lead lots of work or make it complicated just for setting slug from tag name like some comments [here](https://stackoverflow.com/questions/702337/how-to-make-django-slugify-work-properly-with-unicode-strings)

adam commented

@candlerb commented on GitHub (Dec 5, 2019):

Firstly, I think there should be a validation constraint on the tags model and/or table to forbid empty slugs. That would at least prevent getting the database into a broken state.

Next, I note that this problem goes away if tags had to be created before using them (separate feature request #3703). In that case there would be an "Add Tag" page, with a field for the slug, and empty slugs could be rejected before saving them - just like "Add Device Role" currently.

Users would then be forced to create a non-empty slug which is meaningful to them.

(I've also checked what happens when you do a CSV-import of device roles: the "slug" column is mandatory. So there's no problem here either: Netbox does not auto-assign slugs from names)

Now, I have to say unidecode is quite impressive, although it's nearly 2MB of dependency:

>>> unidecode.unidecode("台灣")
'Tai Wan '

It could be exposed as an AJAX completion feature. However, once you've decided to create tags in a form, you'd probably be better off using a Javascript unidecode library in the browser instead.

@candlerb commented on GitHub (Dec 5, 2019): Firstly, I think there should be a validation constraint on the tags model and/or table to forbid empty slugs. That would at least prevent getting the database into a broken state. Next, I note that this problem goes away if tags had to be created before using them (separate feature request #3703). In that case there would be an "Add Tag" page, with a field for the slug, and empty slugs could be rejected before saving them - just like "Add Device Role" currently. ![image](https://user-images.githubusercontent.com/44789/70217761-b5451680-1739-11ea-9ded-7865dbb254d0.png) Users would then be forced to create a non-empty slug which is meaningful to them. (I've also checked what happens when you do a CSV-import of device roles: the "slug" column is mandatory. So there's no problem here either: Netbox does not auto-assign slugs from names) Now, I have to say unidecode is quite impressive, although it's nearly 2MB of dependency: ``` >>> unidecode.unidecode("台灣") 'Tai Wan ' ``` It could be exposed as an AJAX completion feature. However, once you've decided to create tags in a form, you'd probably be better off using a [Javascript unidecode](https://www.npmjs.com/package/unidecode) library in the browser instead.

adam commented

@jeremystretch commented on GitHub (Dec 6, 2019):

After digging into this, I see that there are two separate issues:

Tags can be created on-demand with a blank slug. This is not permitted by the model's field definition, but the manner in which tags are being created escapes this validation.
Unicode characters are effectively being stripped out by Tag's slugify() function assuming the unidecode package is not installed.

As an immediate workaround, I can confirm that installing unidecode will enable valid slugification:

$ pip install unidecode
... 
Installing collected packages: unidecode
Successfully installed unidecode-1.1.1
$ ./manage.py nbshell
i### NetBox interactive shell (jstretch-workstation)
### Python 3.6.8 | Django 2.2.6 | NetBox 2.6.8-dev
### lsmodels() will show available models. Use help(<model>) for more info.
>>> t=Tag()
>>> t.slugify('台灣')
'tai-wan'

(This works because django-taggit attempts to import unidecode and utilizes a no-op wrapper function if that fails.)

I'll work on preventing Tags from being created with blank slugs. But as far as the slugification piece goes, I can't speak to whether the behavior provided by unidecode is valuable.

@jeremystretch commented on GitHub (Dec 6, 2019): After digging into this, I see that there are two separate issues: 1. Tags can be created on-demand with a blank slug. This is *not* permitted by the model's field definition, but the manner in which tags are being created escapes this validation. 2. Unicode characters are effectively being stripped out by Tag's `slugify()` function assuming the `unidecode` package is not installed. As an immediate workaround, I can confirm that installing `unidecode` will enable valid slugification: ``` $ pip install unidecode ... Installing collected packages: unidecode Successfully installed unidecode-1.1.1 $ ./manage.py nbshell i### NetBox interactive shell (jstretch-workstation) ### Python 3.6.8 | Django 2.2.6 | NetBox 2.6.8-dev ### lsmodels() will show available models. Use help(<model>) for more info. >>> t=Tag() >>> t.slugify('台灣') 'tai-wan' ``` (This works because `django-taggit` attempts to import `unidecode` and utilizes a no-op wrapper function if that fails.) I'll work on preventing Tags from being created with blank slugs. But as far as the slugification piece goes, I can't speak to whether the behavior provided by `unidecode` is valuable.

adam commented

@candlerb commented on GitHub (Dec 6, 2019):

Clearly some slug is required, so a default of _1 would work (I believe there is existing logic which cycles _2, _3 etc to ensure uniqueness of slugs) Users could edit them afterwards to make them meaningful, if desired.

@candlerb commented on GitHub (Dec 6, 2019): Clearly *some* slug is required, so a default of `_1` would work (I believe there is existing logic which cycles `_2`, `_3` etc to ensure uniqueness of slugs) Users could edit them afterwards to make them meaningful, if desired.

adam commented

@DanSheps commented on GitHub (Dec 6, 2019):

I think using the unidecode as a slugifier would be the better way to go, IMO, rather then going with a standard incremental index (which we already have on the pk and if we can semi-properly decode unicode, why not?)

I will do some more testing with it, but I think it will be a "good enough" effort on our part to close it out.

It will not handle Japanese, however that is just the nature of the language (uses the same characters as Chinese)

### Python 3.6.8 | Django 2.2.4 | NetBox 2.6.7
### lsmodels() will show available models. Use help(<model>) for more info.
>>>
>>> t=Tag()
>>> t.slugify("元気")
'yuan-qi'
>>> t.slugify("人")
'ren'
>>> t.slugify("一")
'yi'
>>> t.slugify("四")
'si'
>>> t.slugify("市")
'shi'
>>> t.slugify("左")
'zuo'
>>> t.slugify("右")
'you'
>>> t.slugify("川")
'chuan'

Again though, I think this is a good enough approach for something like this.

@DanSheps commented on GitHub (Dec 6, 2019): I think using the unidecode as a slugifier would be the better way to go, IMO, rather then going with a standard incremental index (which we already have on the pk and if we can semi-properly decode unicode, why not?) I will do some more testing with it, but I think it will be a "good enough" effort on our part to close it out. It will not handle Japanese, however that is just the nature of the language (uses the same characters as Chinese) ``` ### Python 3.6.8 | Django 2.2.4 | NetBox 2.6.7 ### lsmodels() will show available models. Use help(<model>) for more info. >>> >>> t=Tag() >>> t.slugify("元気") 'yuan-qi' >>> t.slugify("人") 'ren' >>> t.slugify("一") 'yi' >>> t.slugify("四") 'si' >>> t.slugify("市") 'shi' >>> t.slugify("左") 'zuo' >>> t.slugify("右") 'you' >>> t.slugify("川") 'chuan' ``` Again though, I think this is a good enough approach for something like this.

adam commented

@candlerb commented on GitHub (Dec 6, 2019):

And to be clear, is #3073 likely to be rejected? (Since if #3073 is accepted, this requirement goes away)

@candlerb commented on GitHub (Dec 6, 2019): And to be clear, is #3073 likely to be rejected? (Since if #3073 is accepted, this requirement goes away)

adam commented

@DanSheps commented on GitHub (Dec 7, 2019):

Did you mean a different issue? That one is already closed.

@DanSheps commented on GitHub (Dec 7, 2019): Did you mean a different issue? That one is already closed.

adam commented

@DanSheps commented on GitHub (Dec 7, 2019):

I think you meant #3703

I can't speculate myself, however #3703 would not necessarily negate the need for proper Unicode handling in the Tags so this requirement would likely still be present.

@DanSheps commented on GitHub (Dec 7, 2019): I think you meant #3703 I can't speculate myself, however #3703 would not necessarily negate the need for proper Unicode handling in the Tags so this requirement would likely still be present.

adam commented

@candlerb commented on GitHub (Dec 7, 2019):

Sorry, yes #3703. It would mean that tags would be handled in exactly the same way as device roles, platforms, device types, manufacturers etc: the user would be required to enter a non-empty slug (of their choice) at time of creation of the tag.

@candlerb commented on GitHub (Dec 7, 2019): Sorry, yes #3703. It would mean that tags would be handled in exactly the same way as device roles, platforms, device types, manufacturers etc: the user would be required to enter a non-empty slug (of their choice) at time of creation of the tag.

adam commented

@jeremystretch commented on GitHub (Dec 9, 2019):

To be clear, #3703 is a feature request separate from the issue being addressed here. This needs be resolved regardless of the response to #3703.

It seems we have two options:

Require the installation of unidecode as discussed above, which will return a phonetic (?) representation of the Unicode string.
Allow Unicode characters in the tag slug. Django's slugify function has an option to permit this:

    if allow_unicode:
        value = unicodedata.normalize('NFKC', value)
    else:
        value = unicodedata.normalize('NFKD', value).encode('ascii', 'ignore').decode('ascii')

An example:

>>> slugify('What does 台灣 mean?', allow_unicode=True)
'what-does-台灣-mean'

I don't have a strong opinion either way.

@jeremystretch commented on GitHub (Dec 9, 2019): To be clear, #3703 is a feature request separate from the issue being addressed here. This needs be resolved regardless of the response to #3703. It seems we have two options: 1. Require the installation of `unidecode` as discussed above, which will return a phonetic (?) representation of the Unicode string. 2. Allow Unicode characters in the tag slug. Django's `slugify` function has an option to permit this: ```python if allow_unicode: value = unicodedata.normalize('NFKC', value) else: value = unicodedata.normalize('NFKD', value).encode('ascii', 'ignore').decode('ascii') ``` An example: ```python >>> slugify('What does 台灣 mean?', allow_unicode=True) 'what-does-台灣-mean' ``` I don't have a strong opinion either way.

adam commented

@candlerb commented on GitHub (Dec 9, 2019):

To be clear, #3703 is a feature request separate from the issue being addressed here

Yes...

This needs be resolved regardless of the response to #3703.

...I disagree, because if #3703 is implemented, then it solves this issue at the same time.

Consider for example Device Roles. You can already create a Device Role whose full name consists entirely of high unicode code points. But there is no need to apply unidecode on them to synthesise a slug; you simply cannot save a Device Role with an empty slug. This forces the user to assign a non-empty, ASCII-compatible slug themselves.

#3703 would make tags handled the same. Tags would be created explicitly before use; tags could not spring into existence any other way.

@candlerb commented on GitHub (Dec 9, 2019): > To be clear, #3703 is a feature request separate from the issue being addressed here Yes... > This needs be resolved regardless of the response to #3703. ...I disagree, because if #3703 is implemented, then it solves this issue at the same time. Consider for example Device Roles. You can already create a Device Role whose full name consists entirely of high unicode code points. But there is no need to apply unidecode on them to synthesise a slug; you simply cannot save a Device Role with an empty slug. This forces the user to assign a non-empty, ASCII-compatible slug themselves. #3703 would make tags handled the same. Tags would be created explicitly before use; tags could not spring into existence any other way.

adam commented

@jeremystretch commented on GitHub (Dec 9, 2019):

This forces the user to assign a non-empty, ASCII-compatible slug themselves.

Right, this is a problem. Slugs should be generated automatically regardless. (This is not limited to tags.)

@jeremystretch commented on GitHub (Dec 9, 2019): > This forces the user to assign a non-empty, ASCII-compatible slug themselves. Right, this is a problem. Slugs should be generated automatically regardless. (This is not limited to tags.)

adam commented

@candlerb commented on GitHub (Dec 9, 2019):

Sure, but that's a separate enhancement which ought to have a new ticket. Here you go: #3741

Regarding unicode slugs: they need special encoding for use in URLs. It can be done - but since the main point of slugs is to be embedded in URLs, that's something users will need to be aware of.

@candlerb commented on GitHub (Dec 9, 2019): Sure, but that's a separate enhancement which ought to have a new ticket. Here you go: #3741 Regarding unicode slugs: they need special encoding for use in URLs. It can be done - but since the main point of slugs is to be embedded in URLs, that's something users will need to be aware of.

adam commented

@DanSheps commented on GitHub (Dec 11, 2019):

Another option would be to use urllib.quote() however the slug this would generate would not be user friendly.

@DanSheps commented on GitHub (Dec 11, 2019): Another option would be to use urllib.quote() however the slug this would generate would not be user friendly.

adam commented

@DanSheps commented on GitHub (Jan 7, 2020):

>>> t=Tag()
>>> t.slugify("
>>> t.slugify("つす")
'tsusu'
>>> t.slugify("一つす")
'yi-tsusu'

It looks like hiragana works, but intermixing with kanji you get partial chinese and partial japanese. Any idea's @kobayashi, have you dealt with unicode in your work at all?

@DanSheps commented on GitHub (Jan 7, 2020): ``` >>> t=Tag() >>> t.slugify(" >>> t.slugify("つす") 'tsusu' >>> t.slugify("一つす") 'yi-tsusu' ``` It looks like hiragana works, but intermixing with kanji you get partial chinese and partial japanese. Any idea's @kobayashi, have you dealt with unicode in your work at all?

adam commented

@DanSheps commented on GitHub (Jan 7, 2020):

From PyPI:

On the other hand transliteration (i.e., conveying, in Roman letters, the pronunciation expressed by the text in some other writing system) of languages like Chinese, Japanese or Korean is a very complex issue and this library does not even attempt to address it. It draws the line at context-free character-by-character mapping. So a good rule of thumb is that the further the script you are transliterating is from Latin alphabet, the worse the transliteration will be.

So it looks like unidecode will only transliterate the standard Chinese meaning. I would like to hear from people with experience in other languages, and see if there is a reasonable approximation for most other languages.

@DanSheps commented on GitHub (Jan 7, 2020): From PyPI: > On the other hand transliteration (i.e., conveying, in Roman letters, the pronunciation expressed by the text in some other writing system) of languages like Chinese, Japanese or Korean is a very complex issue and this library does not even attempt to address it. It draws the line at context-free character-by-character mapping. So a good rule of thumb is that the further the script you are transliterating is from Latin alphabet, the worse the transliteration will be. So it looks like unidecode will only transliterate the standard Chinese meaning. I would like to hear from people with experience in other languages, and see if there is a reasonable approximation for most other languages.

adam commented